Welcome to my blog, make yourself at home :)

Fastest Way to Import Big CSV to Redis

Sharing is caring...Share on FacebookShare on Google+Tweet about this on TwitterShare on RedditEmail this to someone

The Problem – Insert millions of keys in Redis

Image you need to load a HUGE (millions of lines) CSV file in a Redis cluster to be accessed by your web server(s). You might quickly think of writing a quick code in your favorite programming language to read the file and insert them in redis. Only to notice it’s taking more than you’ve expected and this is mainly due to the fact that you’re sequentially looping over every line, open a redis connection, run a redis put command, wait for Redis to acknowledge it then close the connection only to open it again in the next iteration.

The Solution – Use redis-cli –pipe

If you can using simple bash scripting skills change the CSV into appropriate Redis commands, you’re done : )

CSV file example:
id, first name, age, gender, nickname, salary
1, John Smith, 40, Male, John, 10000
2, Marco Polo, 43, Male, Marco, 10000
....
1999999, Tom Cruse, 50, Male, Tom, 10001

This could be imported into Redis by converting each line (skipping the first one) to the matching command:
SET "employee_1" "1, John Smith, 40, Male, John, 10000"
SET "employee_2" "2, Marco Polo, 43, Male, Marco, 10000"
....
SET "employee_1999999" "1999999, Tom Cruse, 50, Male, Tom, 10001"

Using the following command:
awk -F, 'NR > 1{ print "SET", "\"employee_"$1"\"", "\""$0"\"" }' file.csv | redis-cli -pipe

Quick Benchmark:

on AWS EC2 – C4.2xlarge & Redis running on M3.Large

  • PHP (simple loop and put implimentation): 7 mins
  • redis-cli (pipeline): 9 secs

Result: more than 40X Times Faster!!!

 

Notes

  • If you started getting a similar error to (ERR unknown command ‘ET’):
    • Then you need to to add “| unix2dos” before “| redis-cli –pipe”
  • you might use the PHP SDK’s in a similar way to pipeline, view here.

See more about this here

0 views

Verify Twitter Digits Session with PHP Laravel on Android

Sharing is caring...Share on FacebookShare on Google+Tweet about this on TwitterShare on RedditEmail this to someone
digits verification step by step

digits verification step by step

0 – Introduction about Fabric & Digits

Twitter announced their Fabric SDK which includes a very interesting service called Digits. Digits allow you to verify a user registration or login via their phone number. Which proves to be a very practical. secure and fast way to authenticate users (specially on mobile). Digits actually handles sending SMS and verification of the codes sent. It currently operates in 200+ countries and 30+ languages.

 1 – Get Current Digits Session

Once a successful SMS verification code has been inserted by the user, the AuthCallback.success method is called, see the following snippet:

DigitsAuthButton digitsButton = (DigitsAuthButton) findViewById(R.id.auth_button);
digitsButton.setCallback(new AuthCallback() {
   @Override
   public void success(DigitsSession session, String phoneNumber) {
       // Do something with the session
       // Verification code goes here
   }

   @Override
   public void failure(DigitsException exception) {
       // Do something on failure
   }
});

2 – Send tokens to server to verify  user login/registration

TwitterAuthConfig authConfig = TwitterCore.getInstance().getAuthConfig();
TwitterAuthToken authToken = session.getAuthToken();
DigitsOAuthSigning oauthSigning = new DigitsOAuthSigning(authConfig, authToken);
Map<String, String> authHeaders = oauthSigning.getOAuthEchoHeadersForVerifyCredentials();

// Add OAuth Echo headers to request
for (Map.Entry<String, String> entry : authHeaders.entrySet()) {
  Log.i("digits", String.format("%s - %s", entry.getKey(), entry.getValue()));
}

This should print something similar:

I/digits﹕ X-Auth-Service-Provider – https://api.digits.com/1.1/sdk/account.json
I/digits﹕ X-Verify-Credentials-Authorization – OAuth oauth_consumer_key=”wHLkT3Z0GiZjWjRbWiopLNW1q”, oauth_nonce=”747094018211629071040851137437457″, oauth_signature=”wdp1vMhafZUYTU%2FMtYjj%2Bc4In2U%3D”, oauth_signature_method=”HMAC-SHA1″, oauth_timestamp=”1437377420″, oauth_token=”3381122110-0YrghJ3EY2PIjEVK8SEmZM7UZ3wnMpexwJTFaVT”, oauth_version=”1.0″

Note: use any network library to send those values to the server; Volley, OkHttp or even HttpsURLConnection

 

3 – Server to verify tokens sent from mobile on server

via CURL:

// set variables with values from previous step
AUTH="whatever value in X-Verify-Credentials-Authorization"
PROVIDER="whatever value in X-Auth-Service-Provider"
// run command
curl -v --get $PROVIDER --header 'Authorization: $AUTH'

 

via PHP:

private function twitterVerifyAuth($provider, $auth) {
    $curl = curl_init();
    curl_setopt($curl,CURLOPT_URL, $provider);
    curl_setopt($curl,CURLOPT_HTTPHEADER, array(
        'Content-length: 0',
        'Content-type: application/json',
        'Authorization: '.$auth,
    ));
    curl_exec($curl);
    $info = curl_getinfo($curl);
    curl_close($curl);
    return $info['http_code'] == 200;
}

 

FAQs:

  1. Don’t have access to session object?
    • Use the following line instead:
      • (TwitterAuthToken) Digits.getInstance().getSessionManager().getActiveSession().getAuthToken();

You’ve any questions, please leave them in the comment section below.

 

0 views

Android ADB No Permissions Fix (Ubuntu)

Sharing is caring...Share on FacebookShare on Google+Tweet about this on TwitterShare on RedditEmail this to someone

The Problem

You connected your device (phone, tablet ..etc ) to your computer via USB to use Android Debugging Bridge (ADB) – to run apps on your device from your computer or a lot of other stuff that you can do using ADB, for more see this – . However, your phone is appearing as “????????????     no permissions”

The Fix (2 solutions)

The Simple Solution (Easy & Not Permanent)

  1. Open Terminal (Shortcut: Ctrl+Alt+T)
  2. Change directory to the Android SDK Tool directory (e.g. /home/mzaher/Downloads/android-sdk-linux_x86/platform-tools)
    1. cd /<your android sdk tools directory location>/android-sdk-linux_x86/platform-tools
  3. Switch to root user and type the root password when asked to.
    1. su
  4. Kill ADB server
    1. ./adb kill-server

Read More →

0 views

How to use SVN with Google code

Sharing is caring...Share on FacebookShare on Google+Tweet about this on TwitterShare on RedditEmail this to someone

1. Download TortoiseSVN
http://tortoisesvn.net/downloads

2. Installing  TortoiseSVN and restart your computer.

3. To get your GoogleCode.com password go to:

http://code.google.com/hosting/settings

svn3

4.You should now see this new items below if you press right-click on any empty place in the explorer or on any folder

svn1

5. Create a New Folder to hold all your files that you will work on.

6. Right-click on this folder and choose “SVN Checkout…”.

Read More →

0 views