Use oplog_replay option to optimize queries on oplog by poudro · Pull Request #3 · rick446/mmm

poudro · 2012-11-07T11:32:54Z

Using an index on the oplog is not very efficient and degrades performance when a big number of writes are being performed in a short amount of time.

Mongo has an internal scheme to perform queries on the oplog that are activated when using the "oplog_replay" option with queries (bit num 3 on http://www.mongodb.org/display/DOCS/Mongo+Wire+Protocol#MongoWireProtocol-OPQUERY).

pymongo doesn't, as yet, allow for the option, although a similar pull-request was proposed for this very purpose.

Please consider applying this patch if pymongo updates the available options in the Cursor class init.

Thx :)

…ce index on capped collections don't work so well...

…ast mark point when mmm crashes

gukoff

It is straight up wrong in so many ways.
Isn't this enough?

q=self._oplog.find(
spec, tailable=True, await_data=True, oplog_replay=True
)

gukoff · 2017-08-15T18:23:06Z

@@ -1,16 +1,16 @@
-#!/usr/bin/env python
+#!/usr/bin/python


This is wrong

gukoff · 2017-08-15T18:26:35Z

+# for name,m in masters.items():
+# pprint.pprint(name)
+# pprint.pprint(m._coll)
+# pprint.pprint('masters output')


There shouldn't be any commented code. If you don't use it, remove it.

gukoff · 2017-08-15T18:33:35Z

- '''Create a bson.Timestamp. Factored into a method so we can delay
- importing socket until gevent.monkey.patch_all() is called. You could
- also insert another timestamp function to set options if you so desire.
- '''


Here is a clear reason not to remove this method.

gukoff · 2017-08-15T18:34:47Z

+ master_uri = self._topology[name_by_id[master['_id']]]['uri']
+ conn = Connection(master_uri)
+ coll = conn.local[MMM_DB_NAME]
+ for cp in coll.find(dict(_id='master_checkpoint')):


Don't use dict for it: {'_id': 'master_checkpoint'}

gukoff · 2017-08-15T18:35:24Z

+ if masters[s._topology[name]['id']] == True:
+ masters[s._topology[name]['id']] = s
+ except:
+ True


What does it mean?

gukoff · 2017-08-15T18:36:41Z

+ try:
+ if masters[s._topology[name]['id']] == True:
+ masters[s._topology[name]['id']] = s
+ except:


except Exception

poudro · 2012-11-07T11:32:54Z

Using an index on the oplog is not very efficient and degrades performance when a big number of writes are being performed in a short amount of time.

Mongo has an internal scheme to perform queries on the oplog that are activated when using the "oplog_replay" option with queries (bit num 3 on http://www.mongodb.org/display/DOCS/Mongo+Wire+Protocol#MongoWireProtocol-OPQUERY).

pymongo doesn't, as yet, allow for the option, although a similar pull-request was proposed for this very purpose.

Please consider applying this patch if pymongo updates the available options in the Cursor class init.

Thx :)

…ce index on capped collections don't work so well...

…ast mark point when mmm crashes

gukoff

It is straight up wrong in so many ways.
Isn't this enough?

q=self._oplog.find(
spec, tailable=True, await_data=True, oplog_replay=True
)

gukoff · 2017-08-15T18:23:06Z

@@ -1,16 +1,16 @@
-#!/usr/bin/env python
+#!/usr/bin/python


This is wrong

gukoff · 2017-08-15T18:26:35Z

+# for name,m in masters.items():
+# pprint.pprint(name)
+# pprint.pprint(m._coll)
+# pprint.pprint('masters output')


There shouldn't be any commented code. If you don't use it, remove it.

gukoff · 2017-08-15T18:33:35Z

- '''Create a bson.Timestamp. Factored into a method so we can delay
- importing socket until gevent.monkey.patch_all() is called. You could
- also insert another timestamp function to set options if you so desire.
- '''


Here is a clear reason not to remove this method.

gukoff · 2017-08-15T18:34:47Z

+ master_uri = self._topology[name_by_id[master['_id']]]['uri']
+ conn = Connection(master_uri)
+ coll = conn.local[MMM_DB_NAME]
+ for cp in coll.find(dict(_id='master_checkpoint')):


Don't use dict for it: {'_id': 'master_checkpoint'}

gukoff · 2017-08-15T18:35:24Z

+ if masters[s._topology[name]['id']] == True:
+ masters[s._topology[name]['id']] = s
+ except:
+ True


What does it mean?

gukoff · 2017-08-15T18:36:41Z

+ try:
+ if masters[s._topology[name]['id']] == True:
+ masters[s._topology[name]['id']] = s
+ except:


except Exception

poudro · 2012-11-07T11:32:54Z

Using an index on the oplog is not very efficient and degrades performance when a big number of writes are being performed in a short amount of time.

Mongo has an internal scheme to perform queries on the oplog that are activated when using the "oplog_replay" option with queries (bit num 3 on http://www.mongodb.org/display/DOCS/Mongo+Wire+Protocol#MongoWireProtocol-OPQUERY).

pymongo doesn't, as yet, allow for the option, although a similar pull-request was proposed for this very purpose.

Please consider applying this patch if pymongo updates the available options in the Cursor class init.

Thx :)

…ce index on capped collections don't work so well...

…ast mark point when mmm crashes

gukoff

It is straight up wrong in so many ways.
Isn't this enough?

q=self._oplog.find(
spec, tailable=True, await_data=True, oplog_replay=True
)

gukoff · 2017-08-15T18:23:06Z

@@ -1,16 +1,16 @@
-#!/usr/bin/env python
+#!/usr/bin/python


This is wrong

gukoff · 2017-08-15T18:26:35Z

+# for name,m in masters.items():
+# pprint.pprint(name)
+# pprint.pprint(m._coll)
+# pprint.pprint('masters output')


There shouldn't be any commented code. If you don't use it, remove it.

gukoff · 2017-08-15T18:33:35Z

- '''Create a bson.Timestamp. Factored into a method so we can delay
- importing socket until gevent.monkey.patch_all() is called. You could
- also insert another timestamp function to set options if you so desire.
- '''


Here is a clear reason not to remove this method.

gukoff · 2017-08-15T18:34:47Z

+ master_uri = self._topology[name_by_id[master['_id']]]['uri']
+ conn = Connection(master_uri)
+ coll = conn.local[MMM_DB_NAME]
+ for cp in coll.find(dict(_id='master_checkpoint')):


Don't use dict for it: {'_id': 'master_checkpoint'}

gukoff · 2017-08-15T18:35:24Z

+ if masters[s._topology[name]['id']] == True:
+ masters[s._topology[name]['id']] = s
+ except:
+ True


What does it mean?

gukoff · 2017-08-15T18:36:41Z

+ try:
+ if masters[s._topology[name]['id']] == True:
+ masters[s._topology[name]['id']] = s
+ except:


except Exception

poudro · 2012-11-07T11:32:54Z

Using an index on the oplog is not very efficient and degrades performance when a big number of writes are being performed in a short amount of time.

Mongo has an internal scheme to perform queries on the oplog that are activated when using the "oplog_replay" option with queries (bit num 3 on http://www.mongodb.org/display/DOCS/Mongo+Wire+Protocol#MongoWireProtocol-OPQUERY).

pymongo doesn't, as yet, allow for the option, although a similar pull-request was proposed for this very purpose.

Please consider applying this patch if pymongo updates the available options in the Cursor class init.

Thx :)

…ce index on capped collections don't work so well...

…ast mark point when mmm crashes

gukoff

It is straight up wrong in so many ways.
Isn't this enough?

q=self._oplog.find(
spec, tailable=True, await_data=True, oplog_replay=True
)

gukoff · 2017-08-15T18:23:06Z

@@ -1,16 +1,16 @@
-#!/usr/bin/env python
+#!/usr/bin/python


This is wrong

gukoff · 2017-08-15T18:26:35Z

+# for name,m in masters.items():
+# pprint.pprint(name)
+# pprint.pprint(m._coll)
+# pprint.pprint('masters output')


There shouldn't be any commented code. If you don't use it, remove it.

gukoff · 2017-08-15T18:33:35Z

- '''Create a bson.Timestamp. Factored into a method so we can delay
- importing socket until gevent.monkey.patch_all() is called. You could
- also insert another timestamp function to set options if you so desire.
- '''


Here is a clear reason not to remove this method.

gukoff · 2017-08-15T18:34:47Z

+ master_uri = self._topology[name_by_id[master['_id']]]['uri']
+ conn = Connection(master_uri)
+ coll = conn.local[MMM_DB_NAME]
+ for cp in coll.find(dict(_id='master_checkpoint')):


Don't use dict for it: {'_id': 'master_checkpoint'}

gukoff · 2017-08-15T18:35:24Z

+ if masters[s._topology[name]['id']] == True:
+ masters[s._topology[name]['id']] = s
+ except:
+ True


What does it mean?

gukoff · 2017-08-15T18:36:41Z

+ try:
+ if masters[s._topology[name]['id']] == True:
+ masters[s._topology[name]['id']] = s
+ except:


except Exception

poudro · 2012-11-07T11:32:54Z

Using an index on the oplog is not very efficient and degrades performance when a big number of writes are being performed in a short amount of time.

Mongo has an internal scheme to perform queries on the oplog that are activated when using the "oplog_replay" option with queries (bit num 3 on http://www.mongodb.org/display/DOCS/Mongo+Wire+Protocol#MongoWireProtocol-OPQUERY).

pymongo doesn't, as yet, allow for the option, although a similar pull-request was proposed for this very purpose.

Please consider applying this patch if pymongo updates the available options in the Cursor class init.

Thx :)

…ce index on capped collections don't work so well...

…ast mark point when mmm crashes

gukoff

It is straight up wrong in so many ways.
Isn't this enough?

q=self._oplog.find(
spec, tailable=True, await_data=True, oplog_replay=True
)

gukoff · 2017-08-15T18:23:06Z

@@ -1,16 +1,16 @@
-#!/usr/bin/env python
+#!/usr/bin/python


This is wrong

gukoff · 2017-08-15T18:26:35Z

+# for name,m in masters.items():
+# pprint.pprint(name)
+# pprint.pprint(m._coll)
+# pprint.pprint('masters output')


There shouldn't be any commented code. If you don't use it, remove it.

gukoff · 2017-08-15T18:33:35Z

- '''Create a bson.Timestamp. Factored into a method so we can delay
- importing socket until gevent.monkey.patch_all() is called. You could
- also insert another timestamp function to set options if you so desire.
- '''


Here is a clear reason not to remove this method.

gukoff · 2017-08-15T18:34:47Z

+ master_uri = self._topology[name_by_id[master['_id']]]['uri']
+ conn = Connection(master_uri)
+ coll = conn.local[MMM_DB_NAME]
+ for cp in coll.find(dict(_id='master_checkpoint')):


Don't use dict for it: {'_id': 'master_checkpoint'}

gukoff · 2017-08-15T18:35:24Z

+ if masters[s._topology[name]['id']] == True:
+ masters[s._topology[name]['id']] = s
+ except:
+ True


What does it mean?

gukoff · 2017-08-15T18:36:41Z

+ try:
+ if masters[s._topology[name]['id']] == True:
+ masters[s._topology[name]['id']] = s
+ except:


except Exception

poudro · 2012-11-07T11:32:54Z

Using an index on the oplog is not very efficient and degrades performance when a big number of writes are being performed in a short amount of time.

Mongo has an internal scheme to perform queries on the oplog that are activated when using the "oplog_replay" option with queries (bit num 3 on http://www.mongodb.org/display/DOCS/Mongo+Wire+Protocol#MongoWireProtocol-OPQUERY).

pymongo doesn't, as yet, allow for the option, although a similar pull-request was proposed for this very purpose.

Please consider applying this patch if pymongo updates the available options in the Cursor class init.

Thx :)

…ce index on capped collections don't work so well...

…ast mark point when mmm crashes

gukoff

It is straight up wrong in so many ways.
Isn't this enough?

q=self._oplog.find(
spec, tailable=True, await_data=True, oplog_replay=True
)

gukoff · 2017-08-15T18:23:06Z

@@ -1,16 +1,16 @@
-#!/usr/bin/env python
+#!/usr/bin/python


This is wrong

gukoff · 2017-08-15T18:26:35Z

+# for name,m in masters.items():
+# pprint.pprint(name)
+# pprint.pprint(m._coll)
+# pprint.pprint('masters output')


There shouldn't be any commented code. If you don't use it, remove it.

gukoff · 2017-08-15T18:33:35Z

- '''Create a bson.Timestamp. Factored into a method so we can delay
- importing socket until gevent.monkey.patch_all() is called. You could
- also insert another timestamp function to set options if you so desire.
- '''


Here is a clear reason not to remove this method.

gukoff · 2017-08-15T18:34:47Z

+ master_uri = self._topology[name_by_id[master['_id']]]['uri']
+ conn = Connection(master_uri)
+ coll = conn.local[MMM_DB_NAME]
+ for cp in coll.find(dict(_id='master_checkpoint')):


Don't use dict for it: {'_id': 'master_checkpoint'}

gukoff · 2017-08-15T18:35:24Z

+ if masters[s._topology[name]['id']] == True:
+ masters[s._topology[name]['id']] = s
+ except:
+ True


What does it mean?

gukoff · 2017-08-15T18:36:41Z

+ try:
+ if masters[s._topology[name]['id']] == True:
+ masters[s._topology[name]['id']] = s
+ except:


except Exception

poudro · 2012-11-07T11:32:54Z

Using an index on the oplog is not very efficient and degrades performance when a big number of writes are being performed in a short amount of time.

Mongo has an internal scheme to perform queries on the oplog that are activated when using the "oplog_replay" option with queries (bit num 3 on http://www.mongodb.org/display/DOCS/Mongo+Wire+Protocol#MongoWireProtocol-OPQUERY).

pymongo doesn't, as yet, allow for the option, although a similar pull-request was proposed for this very purpose.

Please consider applying this patch if pymongo updates the available options in the Cursor class init.

Thx :)

…ce index on capped collections don't work so well...

…ast mark point when mmm crashes

gukoff

It is straight up wrong in so many ways.
Isn't this enough?

q=self._oplog.find(
spec, tailable=True, await_data=True, oplog_replay=True
)

gukoff · 2017-08-15T18:23:06Z

@@ -1,16 +1,16 @@
-#!/usr/bin/env python
+#!/usr/bin/python


This is wrong

gukoff · 2017-08-15T18:26:35Z

+# for name,m in masters.items():
+# pprint.pprint(name)
+# pprint.pprint(m._coll)
+# pprint.pprint('masters output')


There shouldn't be any commented code. If you don't use it, remove it.

gukoff · 2017-08-15T18:33:35Z

- '''Create a bson.Timestamp. Factored into a method so we can delay
- importing socket until gevent.monkey.patch_all() is called. You could
- also insert another timestamp function to set options if you so desire.
- '''


Here is a clear reason not to remove this method.

gukoff · 2017-08-15T18:34:47Z

+ master_uri = self._topology[name_by_id[master['_id']]]['uri']
+ conn = Connection(master_uri)
+ coll = conn.local[MMM_DB_NAME]
+ for cp in coll.find(dict(_id='master_checkpoint')):


Don't use dict for it: {'_id': 'master_checkpoint'}

gukoff · 2017-08-15T18:35:24Z

+ if masters[s._topology[name]['id']] == True:
+ masters[s._topology[name]['id']] = s
+ except:
+ True


What does it mean?

gukoff · 2017-08-15T18:36:41Z

+ try:
+ if masters[s._topology[name]['id']] == True:
+ masters[s._topology[name]['id']] = s
+ except:


except Exception

poudro · 2012-11-07T11:32:54Z

Using an index on the oplog is not very efficient and degrades performance when a big number of writes are being performed in a short amount of time.

Mongo has an internal scheme to perform queries on the oplog that are activated when using the "oplog_replay" option with queries (bit num 3 on http://www.mongodb.org/display/DOCS/Mongo+Wire+Protocol#MongoWireProtocol-OPQUERY).

pymongo doesn't, as yet, allow for the option, although a similar pull-request was proposed for this very purpose.

Please consider applying this patch if pymongo updates the available options in the Cursor class init.

Thx :)

…ce index on capped collections don't work so well...

…ast mark point when mmm crashes

gukoff

It is straight up wrong in so many ways.
Isn't this enough?

q=self._oplog.find(
spec, tailable=True, await_data=True, oplog_replay=True
)

gukoff · 2017-08-15T18:23:06Z

@@ -1,16 +1,16 @@
-#!/usr/bin/env python
+#!/usr/bin/python


This is wrong

gukoff · 2017-08-15T18:26:35Z

+# for name,m in masters.items():
+# pprint.pprint(name)
+# pprint.pprint(m._coll)
+# pprint.pprint('masters output')


There shouldn't be any commented code. If you don't use it, remove it.

gukoff · 2017-08-15T18:33:35Z

- '''Create a bson.Timestamp. Factored into a method so we can delay
- importing socket until gevent.monkey.patch_all() is called. You could
- also insert another timestamp function to set options if you so desire.
- '''


Here is a clear reason not to remove this method.

gukoff · 2017-08-15T18:34:47Z

+ master_uri = self._topology[name_by_id[master['_id']]]['uri']
+ conn = Connection(master_uri)
+ coll = conn.local[MMM_DB_NAME]
+ for cp in coll.find(dict(_id='master_checkpoint')):


Don't use dict for it: {'_id': 'master_checkpoint'}

gukoff · 2017-08-15T18:35:24Z

+ if masters[s._topology[name]['id']] == True:
+ masters[s._topology[name]['id']] = s
+ except:
+ True


What does it mean?

gukoff · 2017-08-15T18:36:41Z

+ try:
+ if masters[s._topology[name]['id']] == True:
+ masters[s._topology[name]['id']] = s
+ except:


except Exception

poudroand others added 2 commits November 6, 2012 17:50