UNIT-II

• NOSQL DATA MANAGEMENT

• Introduction to NoSQL , aggregate data models
,aggregates ,key-value and document data models,
relationships, graph databases, schema less
databases ,materialized views, distribution models ,
sharding , master slave replication , peer-peer
replication , sharding and replication , consistency ,
relaxing consistency , version stamps , mapreduce ,
partitioning and combining , composing map-reduce
calculations
 NoSQL
• NoSQL is a database technology different
from MySQL, primarily because it doesn't
involve the Structured Query Language. The
argument about “SQL vs. NoSQL” is the
comparison of relational vs. non-
relational databases.
RDBMS vs No SQL
NO SQL Database Type
 NO SQL Database types
• Key-value stores and wide column stores provide a single means of
accessing data: by primary key.
• Document databases provide the ability to query on any field within a
document.
• The document data model is the most natural and most productive
because it maps directly to objects in modern object-oriented languages.
• Document databases and graph databases can be consistent or eventually
consistent. MongoDB provide tunable consistency. By default, data is
consistent — all writes and reads access the primary copy of the data.
However, read operation against secondary copies.
• Eventually consistent systems provide some advantages for inserts at the
cost of making reads, updates and deletes more complex, In eventually
consistent systems, there is a period of time in which all copies of the data
are not synchronized.
 Types of NoSQL databases-

Types of NoSQL databases-

• There are 4 basic types of NoSQL databases:
• Key-Value Store – It has a Big Hash Table of keys & values
{Example- Riak,Redis,Memched,membase,dbAmazon S3
(Dynamo)}
• Document-based Store- It stores documents made up of
tagged elements. {Example- CouchDB}
• Column-based Store- Each storage block contains data from
only one column, {Example- HBase, Cassandra}
• Graph-based-A network database that uses edges and nodes
to represent and store data. {Example- Neo4J}
 Key-value databases
• Key-value databases are the simplest of the
NoSQL databases.
• The basic data structure is a dictionary or
map. You can store a value, such as an integer,
string, a JSON structure, or an array, along
with a key used to reference that value. .
• Redis, Riak database are examples of key-
value databases.
 Key-Value model
• key-value model :- key-value stores are the most basic type
of non-relational database.
• It is schema less database and every item in the database is
stored as an attribute name/key together with its value.
• It is commonly used for representing unstructured data.
• Data is considered to be inherently opaque to the database
because based on that key data is accesses whatever it is.
• Example of Key/Value model database : Riak and Redis
EXAMPLE
EXAMPLE
EXAMPLE
Key value Pair
RDBMS
Key value pair
Key- Value
 Example
The following is a document, encoded in JSON:
• { "FirstName": "Bob", "Address": "5 Oak St.",
"Hobby": "sailing" }
 Example
• A second document might be encoded in XML
as:
<contact>
<firstname>Bob</firstname> <lastname>Smith</lastname>
<phone type="Cell">(123) 555-0178</phone> <phone type="Work">(890) 555-
0133</phone> <address>
<type>Home</type>
<street1>123 Back St.</street1> <city>Boys</city> <state>AR</state>
<zip>32225</zip>
<country>US</country>
</address>
</contact>
 Document model
• Document model - Document databases store
data in documents. These documents typically
use JSON (JavaScript Object Notation)
structure.
• MongoDB and CouchDB are two popular
document oriented nosql database.
Below is sample of document that contains info from two different
source /relational table.
 Columnar database
• (Columnar NO SQL database ) :- column model
is similar to traditional relational
database(consisting of table and columns) with
one most significant difference : number of
columns is not fixed for each record.
i.e : columns are created for each row rather
than being predefined by the table structure.
• Example ofColumn model database : HBase
and Cassandra
Column Oriented
Example
 Example
• Column database as key value store with
value is stored as key-value subset.
•
 Graph model
• Graph model - Graph databases use graph
data structure to store data.
• It is being used when traversing relationships
are core to the application like social network
connections.
• Neo4j and Giraph are commonly known graph
database.
Example
 Related Terms
• Scale up vs scale out
• Polyglot Persistent
• Distribution model
• Aggregate
Aggregate
 Distribution Models

• Broadly, there are two paths to data distribution:

• Replication and Sharding.
• Replication takes the same data and copies it over
multiple nodes. Sharding puts different data on different
nodes.
• Replication and sharding are orthogonal techniques:
You can use either or both of them.
• Replication comes into two forms:
Master-Slave and Peer-to-Peer.
 DISTRIBUTION MODEL
1)Master-slave replication
2)Peer-to-peer replication.
 Master-slave
• Master-slave: With master-slave distribution, you
replicate data across multiple nodes.
• One node is designated as the master, or primary.
• This master is the authoritative source for the data
and is usually responsible for processing any updates
to that data.
• The other nodes are slaves, or secondary. A replication
process synchronizes the slaves with the master
Master Slave
 Advantage of Master Slave
1) Master-slave replication is most helpful for
scaling when you have a read-intensive
dataset.
2) You can scale horizontally to handle more
read requests by adding more slave nodes
and ensuring that all read requests are
routed to the slaves.
 Advantage of Master Slave
• 3)Second advantage of master-slave replication is
read resilience: Should the master fail, the
slaves can still handle read requests. Again, this is
useful if most of your data access is reads.
• 4)Having slaves as replicates of the master does
speed up recovery after a failure of the master since
a slave can be appointed as new master very quickly
 Peer to peer
• Master-slave replication helps with read scalability but
doesn’t help with scalability of writes. It provides
resilience against failure of a slave, but not of a master.
• Essentially, the master is still a bottleneck and a single
point of failure.
• Peer-to-peer replication attacks these problems by
not having a master.
• All the replicas have equal weight, they can all accept
writes, and the loss of any of them doesn’t prevent
access to the data store.
Peer to Peer
 Peer to Peer: Disadvantage
• The biggest complication is, again, consistency.
• When you can write to two different places,
you run the risk that two people will attempt
to update the same record at the same time—
a write-write conflict.
 Sharding
• Often, a busy data store is busy because
different people are accessing different parts
of the dataset.
• In these circumstances we can support
horizontal scalability by putting different parts
of the data onto different servers—a
technique that is called sharding
 Sharding
• In the ideal case, we have different users all talking to
different server nodes. Each user only has to talk to one
server, so gets rapid responses from that server
• In order to get close to it we have to ensure that data
that’s accessed together is clumped together on the same
node and that these clumps are arranged on the nodes to
provide the best data access.
• HOW ?:This is where aggregate orientation comes in . The
whole point of aggregates is that we design them to
combine data that’s commonly accessed together—so
aggregates leap out as an obvious unit of distribution.
Sharding: Strategy behind Sharding
• 1)If you know that most accesses of certain
aggregates are based on a physical location,
you can place the data close to where it’s being
accessed. Example: If you have orders for
someone who lives in Boston, you can place
that data in your eastern US data center
• 2) Keep load even
• 3) certain days of the week—so there may be
domain-specific rules you’d like to use
Sharding: Strategy behind Sharding
• Bigtable paper [Chang etc.] described keeping
its rows in lexicographic order and sorting web
addresses based on reversed domain names
(e.g., com.martinfowler).
• This way data for multiple pages could be
accessed together to improve processing
efficiency.
 Sharding
• Many NoSQL databases offer auto- sharding, where the
database takes on the responsibility of allocating data to
shards and ensuring that data access goes to the right shard
• Sharding is particularly valuable for performance because it
can improve both read and write performance.
• Using replication, particularly with caching, can greatly
improve read performance but does little for applications that
have a lot of writes.
• Sharding provides a way to horizontally scale writes also.
• Sharding does little to improve resilience when used alone.
• So combine Sharding and Replication
Sharding
Combing Sharding and Replications
 Sharding & Replication both
• Replication and sharding are strategies that can be
combined.
• If we use both master-slave replication and sharding this
means that we have multiple masters, but each data item
only has a single master.
• Depending on your configuration, you may choose a
node to be a master for some data and slaves for others,
or you may dedicate nodes for master or slave duties.
• Using peer-to-peer replication and sharding is a
common strategy for column-family databases.
 Consistency:
• Relational databases try to exhibit strong
consistency

• This issue is called a write-write conflict: two people updating the

same data item at the same time.

• When the writes reach the server, the server will serialize them—
decide to apply one, then the
other. Let’s assume it uses alphabetical order and picks Martin’s
update first, then Pramod’s. Without any concurrency control,
Martin’s update would be applied and immediately overwritten by
Pramod’s. In this case Martin’s is a lost update
• A pessimistic approach works by preventing conflicts from
occurring; an optimistic approach lets conflicts occur, but
detects them and takes action to sort them out.

• For update conflicts, the most common pessimistic

approach is to have write locks, so that in order to change
a value you need to acquire a lock, and the system
ensures that only one client can get a lock at a time.
• One optimistic way to handle a write-write conflict—save both
updates and record that they are in conflict.

• This approach is familiar to many programmers from version

control systems, particularly distributed version control systems
that by their nature will often have conflicting commits.

• The next step again follows from version control: You have to
merge the two updates somehow. Maybe you show both values
to the user and ask them to sort it out
• People first prefer pessimistic concurrency because they are
determined to avoid conflicts but it (Pessimistic approaches) often
severely degrade the responsiveness of a system to the degree that it
becomes unfit for its purpose.

• This problem is made worse by the danger of errors—pessimistic

concurrency often leads to deadlocks, which are hard to prevent and
debug.

• Replication makes it much more likely to run into write-write conflicts

CAP
 CAP
• NoSQL world, phrases such as “CAP theorem” and
“eventual consistency” comes in.

• CAP Theorem: NoSQL world it’s common to refer to

the CAP theorem as the reason why you may need to
relax consistency.

• It was originally proposed by Eric Brewer in 2000 and

given a formal proof by Seth Gilbert and Nancy Lynch
• The basic statement of the CAP theorem is that, given the three
properties of Consistency, Availability, and Partition tolerance, you can
only get two.

• Availability :It means that if you can talk to a node in the cluster, it can
read and write data.

Partition tolerance means that the cluster can survive communication

breakages in the cluster that separate the cluster into multiple
partitions unable to communicate with each other (situation known as
a split brain).
• A single-server system is the obvious example of a CA system—a
system that has Consistency and
Availability but not Partition tolerance.

• A single machine can’t partition, so it does not have to worry

about partition tolerance. There’s only one node—so if it’s up, it’s
available. Being up and keeping consistency is reasonable. This is
the world that most relational database systems live in.

• But clusters have to be tolerant of network partitions

 CAP
• There are cases where you can deal gracefully with inconsistent answers to
requests.
•
• These situations are closely tied to the domain and require domain knowledge to
know how to resolve. Thus you can’t usually look to solve them purely within the
development team—you have to talk to domain experts. :Examples:

• Hotel booking / shopping cart :Inconsistency vs availability( Availability is more

important then Inconsistency)
• Financial Transaction: consistency must
• Posting News Item to Media: can tolerate Inconsistency .
• Thus Different data items may have different tolerances for staleness
 BASE
• Advocates of NoSQL often say that instead of
following the ACID properties of relational
transactions, NoSQL systems follow the BASE
properties (Basically Available, Soft state,
Eventual consistency)
 ACID
• The ACID Consistency Model :Many developers are familiar with ACID transactions
from working with relational databases.

• The ACID acronym stands for:

Atomic
– All operations in a transaction succeed or every operation is rolled back.
• Consistent
– On the completion of a transaction, the database is structurally sound.
• Isolated
– Transactions do not contend with one another. Contentious access to data is
moderated by the database so that transactions appear to run sequentially.
• Durable
– The results of applying a transaction are permanent, even in the presence of
failures.
• ACID properties mean that once a transaction is complete, its data is consistent (tech
lingo: write consistency) and stable on disk, which may involve multiple distinct
memory locations.
 BASE
The BASE Consistency Model
• For many domains and use cases, ACID transactions are far more pessimistic
(i.e., they’re more worried about data safety) than the domain actually
requires.
• In the NoSQL world, ACID transactions are less fashionable as some databases
have loosened the requirements for immediate consistency, data freshness
and accuracy in order to gain other benefits, like scale and resilience.
• Basic Availability: The database appears to work most of the time.
• Soft-state: Stores don’t have to be write-consistent, nor do different replicas
have to be mutually consistent all the time.
• Eventual consistency: Stores exhibit consistency at some later point (e.g., lazily
at read time).
• BASE properties are much looser than ACID guarantees, but there isn’t a direct
one-for-one mapping between the two consistency models.
The BASE consistency model is primarily used by aggregate stores, including
column family, Key value and document Stores.
 Version Stamps
• Many critics of NoSQL databases focus on the lack of support for
transactions.

• Transactions are a useful tool that helps programmers support

consistency.

• One reason why many NoSQL supporters worry less about a lack of
transactions is that aggregate-oriented
NoSQL databases do support atomic updates within an aggregate—and
aggregates are designed so that their data forms a natural unit of up
date.

• Version Stamps
 Version Stamps
• A good way managing Transaction and Consistency
problem is to ensure that records in the database
contain some form of version stamp

• Version stamp : a field that changes every time the

underlying data in the record changes. When you read
the data you keep a note of the version stamp, so that
when you write data you can check to see if the
version has changed.
Conditional update :compare-and-set (CAS)

• Some databases provide a mechanism of conditional

update that allows you to ensure updates won’t be
based on stale data.

• Read and update. (Sometimes this is called a compare-

and-set (CAS) operation.

• In database CAS is a conditional update that compares

a version stamp of the value.
Approaches to generate Version Stamp
1) You can use a counter, always incrementing it when you update the
resource. Counters are useful since they make it easy to tell if one version
is more recent than another. On the other hand, they require the server to
generate the counter value, and also need a single master to ensure the
counters aren’t duplicated.

2) Another approach is to create a GUID, ( globally unique ID)a large

random number that’s guaranteed to be unique. These use some
combination of dates, hardware information, and whatever other sources
of randomness they can pick up. The nice thing about GUIDs is that they
can be generated by anyone and you’ll never get a duplicate; a
disadvantage is that they are large and can’t be compared directly for
recentness.
Approaches to generate Version Stamp
• 3) A third approach is to make a hash of the contents of the resource. With a big
enough hash key size, a content hash can be globally unique like a GUID and can
also be generated by anyone; the advantage is that they are deterministic—any
node will generate the same content hash for same resource data. However, like
GUIDs they can’t be directly compared for recentness, and they can be lengthy.

• 4) A fourth approach is to use the timestamp of the last update. Like counters,
they are reasonably short and can be directly compared for recentness, yet have
the advantage of not needing a single Ma ster. Multiple machines can generate
timestamps—but to work properly, their clocks have to be kept in sync. One
node with a bad clock can cause all sorts of data corruptions. There’s also a
danger that if the timestamp is too granular you can get duplicates—it’s no good
using timestamps of a millisecond precision if you get many updates per
millisecond.
 Version Stamp
• You can blend the advantages of these different
version stamp schemes by using more than one
of the m to create a composite stamp. For
example, CouchDB uses a combination of
counter and content hash.
• Version Stamp helps to avoid update conflicts, it
is also useful for providing session consistency
 Version Stamp :vector stamp
• A vector stamp is a set of counters, one for each
node.
• A vector stamp for three nodes (blue, green, black)
would look something like:
• [blue: 43,green: 54, black: 12].
• Each time a node has an internal update, it updates
its own counter, so an up date in the green node
would change the vector to [blue: 43, green: 55, black:
12].
 Conclude: Version stamps
• Version stamps help you detect concurrency conflicts. When you
read data, then update it, you can check the version stamp to
ensure nobody updated the data between your read and write.

• Version stamps can be implemented using counters, GUIDs,

content hashes, timestamps, or a combination of these.

• With distributed systems, a vector of version stamps allows

you to detect when different nodes have conflicting updates.
Polyglot Persistence
Polygot persistence: The point of view of using different data stores in
different circumstances is known as Polyglot Persistence.
Today, most large companies are using a variety of different data storage
technologies for different kinds of data. Many companies still use relational
databases to store some data, but the persistence needs of applications
are evolving from predominantly relational to a mixture of data sources.
Polyglot persistence is commonly used to define this hybrid approach. The
definition of polyglot is “someone who speaks or writes several languages.”
The term polyglot is redefined for big data as a set of applications that use
several core database technologies.
 Job Tracker

• The JobTracker is the service within Hadoop that farms out MapReduce tasks to specific
nodes in the cluster, ideally the nodes that have the data, or at least are in the same rack.
• Client applications submit jobs to the Job tracker.
• The JobTracker talks to the NameNode to determine the location of the data
• The JobTracker locates TaskTracker nodes with available slots at or near the data
• The JobTracker submits the work to the chosen TaskTracker nodes.
• The TaskTracker nodes are monitored. If they do not submit heartbeat signals often
enough, they are deemed to have failed and the work is scheduled on a different
TaskTracker.
• A TaskTracker will notify the JobTracker when a task fails. The JobTracker decides what to do
then: it may resubmit the job elsewhere, it may mark that specific record as something to
avoid, and it may even blacklist the TaskTracker as unreliable.
• When the work is completed, the JobTracker updates its status.
• Client applications can poll the JobTracker for information.
• The JobTracker is a point of failure for the Hadoop MapReduce service. If it goes down, all
running jobs are halted.
 Map-Reduce
• The rise of aggregate-oriented databases is in large part due to the growth of
clusters. Running on a cluster means you have to make your tradeoffs in data storage
differently than when running on a single machine.
• Clusters don’t just change the rules for data storage—they also change the rules for
computation. If you store lots of data on a cluster, processing that data efficiently
means you have to think differently about how you organize your processing.
• With a centralized database, there are generally two ways you can run the processing
logic against it: either on the database server itself or on a client machine.
• Running it on a client machine gives you more flexibility in choosing a programming
environment, which usually makes for programs that are easier to create or extend.
This comes at the cost of having to shift lots of data from the database server.
• If you need to hit a lot of data, then it makes sense to do the processing on the
server, paying the price in programming convenience and increasing the load on the
database server.
 Need for Map Reduce
• When you have a cluster, there is good news
immediately—you have lots of machines to
spread the computation over.
• However, you also still need to try to reduce
the amount of data that needs to be
transferred across the network by doing as
much processing as you can on the same node
as the data it needs.
 Map-Reduce
• The map-reduce pattern (a form of Scatter-Gather )is a way to organize
processing in such a way as to take advantage of multiple machines on a
cluster while keeping as much processing and the data it needs
together on the same machine.
• Example: Order & customer
• Orders can be aggregated together but to get
• “Product Revenue Report”: you’ll have to visit every machine in the
cluster and examine many records on each machine to get the Product
revenue Report
• So go for MAP REDUCE
Map reduce with single Reduce task
 MAP
• The first stage in a map-reduce job is the map.
• A map is a function whose input is a single
aggregate and whose output is a bunch of key
value pairs.
• In this case, the input would be an order. The
output would be key-value pairs corresponding
to each items. Each one would have the product
ID as the key and an embedded map with the
quantity and price as the values
MAP
Reduce
Map Reduce
 Map
• Each application of the map function is
independent of all the others.
• This allows them to be safely parallelizable, so
that a map-reduce framework can create
efficient map tasks on each node and freely
allocate each order to a map task. This yields a
great deal of parallelism and locality of data
access.
 Map Reduce
• A map operation only operates on a single record;
the reduce function takes multiple map outputs
with the same key and combines their values. So, a
map function might yield 1000 line items from
orders; then reduce function would reduce down to
one, with the totals for the quantity and revenue.
While the map function is limited to working only
on data from a single aggregate, the reduce
function can use all values emitted for a single key
 Partitioning and Combining
• In the simplest form, we think of a map-
reduce job as having a single reduce function.
The outputs from all the map tasks running on
the various nodes are concatenated together
and sent into the reduce.
• While this will work, there are things we can
do to increase the parallelism and to reduce
the data transfer
 Partitioning and Combining
• The first thing we can do is increase parallelism by partitioning the output of the
mappers.
• Each reduce function operates on the results of a single key.
• This is a limitation—it means you can’t do anything in the reduce that operates
across keys—but it’s also a benefit in that it allows you to run multiple reducers
in parallel.
• To take advantage of this, the results of the mapper are divided up based the key
on each processing node.
• Typically, multiple keys are grouped together into partitions.
• The framework then takes the data from all the nodes for one partition,
combines it into a single group for that partition, and sends it off to a reducer.
• Multiple reducers can then operate on the partitions in parallel, with the final
results merged together. (This step is also called “shuffling,” and the partitions
are sometimes referred to as “buckets” or “regions.”)
 Partitioning and Combining
• The next problem we can deal with is the amount of
data being moved from node to node between the map
and reduce stages.
• Much of this data is repetitive, consisting of multiple
key-value pairs for the same key.
• A combiner function cuts this data down by combining
all the data for the same key into a single value .
• A combiner function is, in essence, a reducer function
—indeed, in many cases the same function can be used
for combining as the final reduction.
Map- Reduce dataflow with Multiple Reduce
task
 Composing Map-Reduce Calculations

• The map-reduce approach is a way of thinking about

concurrent processing that trades off flexibility in how you
structure your computation for a relatively straightforward
model for parallelizing the computation over a cluster.
• Since it’s a tradeoff, there are constraints on what you can
do in your calculations.
• Within a map task, you can only operate on a single
aggregate. Within a reduce task, you can only operate on a
single key.
• This means you have to think differently about structuring
your programs so they work well within these constraints.
 Composing Map-Reduce Calculations

• One simple limitation is that you have to

structure your calculations around operations
that fit in well with the notion of a reduce
operation.
Composing Map-Reduce Calculations