Saturday, May 26, 2012

MemCacheD >>>>>>>>>>>>>>>>


What is MemCacheD?
Memcached is a general-purpose distributed memory caching system that was originally developed by Danga Interactive for Live Journal, but is now used by many other sites.
It is often used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of times an external data source (such as a database or API) must be read. Memcached runs on Unix, Linux, Windows and MacOSX and is distributed under a permissive free software license.

Memcached's APIs provide a giant hash table distributed across multiple machines. When the table is full, subsequent inserts cause older data to be purged in least recently used (LRU) order. Applications using Memcached typically layer requests and additions into RAM before falling back on a slower backing store, such as a database.


How Does It Work ?
Memcached is similar to a hash table. A hash table is implemented as an array of buckets. Each bucket (array element) contains a list of nodes, with each node containing [key, value]. This list later is searched to find the node containing the right key. Most hashes start small and dynamically resize over time as the lists of the buckets get too long. A request to get/set a key with a value requires that the key be run through a hash function. A hash function is a one-way function mapping a key (be it numeric or string) to some number that is going to be the bucket number. Once the bucket number has been calculated, the list of nodes for that bucket is searched, looking for the node with the given key. If it's not found, a new one can be added to the list.

Memcached represents the user with a dictionary interface but instead of a one way hash function it represents a two way hash. The first kind of hashing is done in the client library where the client decides to which server to send the request to. It does so by hashing the key into a list of virtual buckets, with each bucket representing a Memcached server. Each Memcached server represents a typical hash table.  Each Memcached server instance is independent, unaware of the presence or status of other servers. All these servers are unified together by the client library. If a server fails, the clients can be configured to route around the dead machine or machines and use the remaining active servers.

The system is used by sites including YouTube, Facebook, and Twitter.
Architecture:
The system uses a client–server architecture. The servers maintain a key–value associative array; the clients populate this array and query it. Keys are up to 250 bytes long and values can be at most 1 megabyte in size.
Clients use client side libraries to contact the servers which, by default, expose their service at port 11211. Each client knows all servers; the servers do not communicate with each other. If a client wishes to set or read the value corresponding to a certain key, the client's library first computes a hash of the key to determine the server that will be used. Then it contacts that server. The server will compute a second hash of the key to determine where to store or read the corresponding value.
The servers keep the values in RAM; if a server runs out of RAM, it discards the oldest values. Therefore, clients must treat Memcached as a transitory cache; they cannot assume that data stored in Memcached is still there when they need it. MemcacheDB and Membase provide persistent storage while maintaining memcached protocol compatibility.
If all client libraries use the same hashing algorithm to determine servers, then clients can read each other's cached data; this is obviously desirable.
A typical deployment will have several servers and many clients. However, it is possible to use Memcached on a single computer, acting simultaneously as client and server.

Example pseudo code:

Note that all functions described on this page are pseudo-code only. Memcached calls and programming languages may vary based on the API used.
Converting database or object creation queries to use Memcached is simple. Typically, when using straight database queries, example code would be as follows:

 declare function get_data(userid as xs:int) {
let   $data: = fn:collection(“userData”)/users/user[@userId=userid];
                  return data;
 };
  
After conversion to Memcached, the same call might look like the following
 
declare function get_data(userid as xs:int) {
    /* first try the cache */
 let  $data:= memcached_fetch("user:" + userid);
    if (!data) {
      data = fn:collection(“userData”)/users/user[@userId=userid];
                  return data;
     }
 };
 
The client would first check whether a Memcached value with the unique key "userid" exists, where userid is some number. If the result does not exist, it would select from the database as usual, and set the unique key using the Memcached API add function call.
However, if only this API call were modified, the server would end up fetching incorrect data following any database update actions: the Memcached "view" of the data would become out of date. Therefore, in addition to creating an "add" call, an update call would also be needed using the Memcached set function.
 declare function update_data(userid as xs:int, updateString as xs:string) {
   /* first update db */
   let  $result: = execute(updateString);
    if (result) {
       /* database update successful : fetch data to be stored in cache */
      let  $data: = fn:collection(“userData”)/users/user[@userId=userid];
       /* then store in cache until next get */                          
     let $expireTime:=0;
       memcached_set(userid, data,expireTime);
    }
 };
This call would update the currently cached data to match the new data in the database, assuming the database query succeeds. An alternative approach would be to invalidate the cache with the Memcached delete function, so that subsequent fetches result in a cache miss. Similar action would need to be taken when database records were deleted, to maintain either a correct or incomplete cache.

Installing and Using Mem-cached with JAVA:
Download memcached exe:  http://allegiance.chi-town.com/MemCacheDManager.aspx[installer]
                                                  Or http://code.jellycan.com/memcached/










 Note:
 Original: for Linux.
 Win32 binary: for windows cmd prompt based.

Dependencies: .net framework (for windows only, if using GUI version)


JAVADOC Memcached:

Reference:

After downloading all the stuffs, let’s learn how to use it.
Step1- Install Memcached GUI server.
Step2- Run Memcached [See first look below.]


  
Step3- configure memcached server.
Ø  Click on “Add Server” button.


Ø  Provide “Server name”, Username & password of your operating system, if you want to use.
           

Ø  Remember server name should be ‘localhost’ or IP.

  
Ø  Click on “Add instance” button. [You will see this window]



Ø  Provide “Instance Name”, “Port no. (if want any specific, otherwise leave it as it is default.)”.
Ø  Provide “IP” if want to use some specific IP.
Ø  Change memory settings, if required.
Ø   Add maximum no. of connection to allow on the server.
Ø  Now click on”Apply button”.
Ø  It will ask to start your instance of server, and then click on “Yes”.

Ø  Instance will be created successfully.
Ø  We can add n number of instances as required; can be run on different-2 IP and Ports.
Ø  Step3- Now we can see the status of our instance.

Ø  Step4- Now it’s time to use it.
1-      Now add the “memcached-2.5.jar” in class path.

JAVA requirements:
Ø   You can use MemcachedClient.set () to store a simple string or a complex object. When you store a complex object, MemcachedClient will first serialize the object and then store it. As a result, every object that you store in Memcached must be serializable, and the key must also be a string. The set () method returns an object of Future. When we call the set method, it is executed asynchronously, so the control moves to the next line without waiting for a response from the Memcached server. If you needed to know the result of the set operation, then you would call setResult.get () on the Future object instance.
Ø  If you have stored complex objects then the client will first retrieve the value and then deserialize it and return the object.

JAVA CODE:
package memcached;
import java.io.Serializable;

public class ObjectModel implements Serializable{

                private static final long serialVersionUID = 1L;
                String name;
                String address;
                int age;
                public ObjectModel(String name, String address, int age) {
                                super();
                                this.name = name;
                                this.address = address;
                                this.age = age;
                }
                public String getName() {
                                return name;
                }
                public void setName(String name) {
                                this.name = name;
                }
                public String getAddress() {
                                return address;
                }
                public void setAddress(String address) {
                                this.address = address;
                }
                public int getAge() {
                                return age;
                }
                public void setAge(int age) {
                                this.age = age;
                }
                public String toString() {
                                return name+" @ "+age+" @ "+address;
                }
}


package memcached;

import java.io.IOException;
import java.util.Scanner;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
import net.spy.memcached.AddrUtil;
import net.spy.memcached.MemcachedClient;

/**
* The Class MemcachedTestClient.
*
* @author Abhinav
*/
public class MemcachedTestClient {
public static void main(String[] args) throws IOException,
InterruptedException, ExecutionException {
UserModel objMdl = new UserModel("abhinav", "noida", 25);
String commandName = "";
String keyName = "";
String value = "";
int expTimeInSeconds = 0;
long casIdentifier = 0;

Scanner sc = new Scanner(System.in);
System.out.println("Enter command: ");
commandName = sc.nextLine();
System.out.println("Enter keyName: ");
keyName = sc.nextLine();
if (!(commandName.equals("get") || commandName.equals("delete") || commandName
.equalsIgnoreCase("getObject"))) {
if (!(commandName.equalsIgnoreCase("addObject"))) {
System.out.println("Enter value: ");
value = sc.nextLine();
}
if (commandName.equals("append") || commandName.equals("prepend")) {
System.out.println("Enter casIdentifier: ");
casIdentifier = sc.nextInt();
} else {
System.out.println("Enter expiryTime(in seconds): ");
expTimeInSeconds = sc.nextInt();
}
}
MemcachedClient memcachedClient = new MemcachedClient(AddrUtil
.getAddresses("localhost:11211"));// getting connection from
// server

long startTime = System.nanoTime();
if (commandName.equals("get")) {
/**
* It is used to get the value of the key from a Memcached server.
* If you have stored complex objects then the client will first
* retrieve the value and then deserialize it and return the object.
* The get() method returns null if the value is not found or is
* expired.
*/
System.out.println("Key Name: " + keyName);
System.out.println("Value of key: " + memcachedClient.get(keyName));
} else if (commandName.equals("set")) {
/**
* MemcachedClient.set()
*
* You can use MemcachedClient.set() to store a simple string or a
* complex object. When you store a complex object, MemcachedClient
* will first serialize the object and then store it. As a result,
* every object that you store in Memcached must be serializable,
* and the key must also be a string. The set() method returns an
* object of Future. When we call the set method, it is executed
* asynchronously, so the control moves to the next line without
* waiting for a response from the Memcached server. If you needed
* to know the result of the set operation, then you would call
* setResult.get() on the Future object instance.
*
* @param: String keyName
* @param: int expiration value (i.e, the expiration of cached
*         object in seconds, default is 0 seconds, it means cached
*         object will not expire until we delete it.)
* @param: value
* */
System.out.println("Key Name: " + keyName + " value: " + value
+ " expTimeInSeconds of this object: " + expTimeInSeconds);
Future<Boolean> result = memcachedClient.set(keyName,
expTimeInSeconds, value);
System.out.println("Result of set operation: " + result.get());
} else if (commandName.equals("add")) {
/**
* It is used to add an object to the cache only if it does not
* already exist. Any complex object will be serialized before
* adding to cache. So every object that you store in Memcached must
* be serializable.
*
* @param: String keyName
* @param: int expiration value (i.e, the expiration of cached
*         object in seconds, default is 0 seconds, it means cached
*         object will not expire until we delete it.)
* @param: value
*/
System.out.println("Key Name: " + keyName + " value: " + value
+ " expTimeInSeconds of this object: " + expTimeInSeconds);

Future<Boolean> result = memcachedClient.add(keyName,
expTimeInSeconds, value);
System.out.println("Result of add operation: " + result.get());
} else if (commandName.equals("replace")) {
/**
* It replaces an object with the value for the given key (if there
* is already such a value).
*
* @param: String keyName
* @param: int expiration value (i.e, the expiration of cached
*         object in seconds, default is 0 seconds, it means cached
*         object will not expire until we delete it.)
* @param: value
*/

System.out.println("Key Name: " + keyName + " value: " + value
+ " expTimeInSeconds of this object: " + expTimeInSeconds);
Future<Boolean> result = memcachedClient.replace(keyName,
expTimeInSeconds, value);
System.out.println("Result of replace operation: " + result.get());
} else if (commandName.equals("delete")) {
/**
* It deletes a given key from the cache.
* */
System.out.println("Key Name: " + keyName);
Future<Boolean> result = memcachedClient.delete(keyName);
System.out.println("Result of delete operation: " + result.get());
} else if (commandName.equals("append")) {
/**
* Append (add value at last position of your previous value )to an
* existing value in the cache. prepend
*
* @param: long casIdentifier
* @param: String keyName
* @param: value
* */

System.out.println("Key Name: " + keyName + " value: " + value
+ " casIdentifier of this object: " + casIdentifier);
Future<Boolean> result = memcachedClient.append(casIdentifier,
keyName, value);
System.out.println("Result of append operation: " + result.get());
} else if (commandName.equals("prepend")) {

/**
* Prepend(add value at start position of your previous value) from
* an existing value in the cache.
*
* @param: long casIdentifier
* @param: String keyName
* @param: value
* */

System.out.println("Key Name: " + keyName + " value: " + value
+ " casIdentifier of this object: " + casIdentifier);
Future<Boolean> result = memcachedClient.prepend(casIdentifier,
keyName, value);
System.out.println("Result of prepend operation: " + result.get());
} else if (commandName.equals("addObject")) {
/**
* It is used to add an object to the cache only if it does not
* already exist. Any complex object will be serialized before
* adding to cache. So every object that you store in Memcached must
* be serializable.
*
* @param: String keyName
* @param: int expiration value (i.e, the expiration of cached
*         object in seconds, default is 0 seconds, it means cached
*         object will not expire until we delete it.)
* @param: value
*/
System.out.println("Key Name: " + keyName
+ " expTimeInSeconds of this object: " + expTimeInSeconds);

Future<Boolean> result = memcachedClient.add(keyName,
expTimeInSeconds, objMdl);
System.out
.println("Result of addObject operation: " + result.get());
} else if (commandName.equalsIgnoreCase("getObject")) {
/**
* It is used to get the value of the key from a Memcached server.
* If you have stored complex objects then the client will first
* retrieve the value and then deserialize it and return the object.
* The get() method returns null if the value is not found or is
* expired.
*/
System.out.println("Key Name: " + keyName);
System.out
.println("Object of key: " + memcachedClient.get(keyName));
} else {
System.out.println("Command not found.");
}

long endTime = System.nanoTime();
System.out.println("Total elapsed time in nanoseconds: "
+ (endTime - startTime));
memcachedClient.shutdown();
}
}

 

 

Output>>
Add Object>>>>
Enter command:
addObject
Enter keyName:
abhinav
Enter expiryTime(in seconds):
120
2012-05-29 15:52:12.820 INFO Key Name: abhinav expTimeInSeconds of this object: 120
net.spy.memcached.MemcachedConnection:  Added {QA sa=localhost/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2012-05-29 15:52:12.820 INFO net.spy.memcached.MemcachedConnection:  Connection state changed for sun.nio.ch.SelectionKeyImpl@9971ad
Result of addObject operation: true
Total elapsed time in nanoseconds: 21200869
2012-05-29 15:52:12.851 INFO net.spy.memcached.MemcachedClient:  Shut down memcached client

Get Object>>>>
Enter command:
getObject
Enter keyName:
abhinav
2012-05-29 15:52:31.883 INFO net.spy.memcached.MemcachedConnection:  Added {QA sa=localhost/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
Key Name: abhinav
2012-05-29 15:52:31.915 INFO net.spy.memcached.MemcachedConnection:  Connection state changed for sun.nio.ch.SelectionKeyImpl@9971ad
Object of key: abhinav @ 25 @ noida
Total elapsed time in nanoseconds: 24881729
2012-05-29 15:52:31.915 INFO net.spy.memcached.MemcachedClient:  Shut down memcached client

Starting, Stopping, restarting and deleting server instance:
Ø  Right click on any instance, click Stop Service [to stop], Restart Service [to restart] ,Start service [to start] and Delete instance [to delete].


Add another instance,re-deployee/update memcached and Delete Server:
Ø  Right click on server, click Add New Instance Service [to add new instance server], Redeploy/Update MemCacheD [to redeploy or update], Delete Server [to delete the server] and Refresh [to refresh the server list].

We can also use Binary version of this server:
Ø  Download the binary version as indicated link above.
Ø  Extract zip file.
Ø  Just run the “memcached.exe” file.
It looks like this:



Ø  We can now use it directly.
Ø  It is default configured on localhost at port no. “11211” with default RAM memory as 64 megabyte.


References: http://en.wikipedia.org/wiki/Memcached



Leave your comments/suggestions below, Otherwise the next time :)