large negative increment values in line profiler

[nathantsoi]

i am using memory profile as follows:

an_instance = AClass()
precision=1
backend='psutil'
with open("mem-profile.txt", 'w') as stream:
 prof = mp.LineProfiler(backend=backend)
 res = prof(an_instance.window)(template)
 mp.show_results(prof, stream=stream, precision=precision)
return res

however the results seem strange, I see large negative increments, much larger than any calls with positive increments:

Line # Mem usage Increment
================================
 51 192.2 MiB 192.2 MiB
 52 
 53 
 54 
 55 
 56 
 57 
 58 192.2 MiB 0.0 MiB
 59 736.4 MiB -8.1 MiB
 60 736.4 MiB -6.8 MiB
 61 736.4 MiB -7.4 MiB
 62 736.4 MiB -8.1 MiB
 63 
 64 736.4 MiB -7.7 MiB
 65 736.4 MiB -8.0 MiB
 66 736.4 MiB -7.9 MiB
 67 736.4 MiB -8.1 MiB
 68 
 69 736.4 MiB -7.9 MiB
 70 736.4 MiB -8.0 MiB
 71 736.4 MiB -8.1 MiB
 72 736.4 MiB -8.1 MiB
 73 
 74 736.4 MiB -8.1 MiB
 75 736.4 MiB -8.1 MiB
 76 736.4 MiB -8.1 MiB
 77 
 78 
 79 
 80 
 81 736.4 MiB -47.7 MiB
 82 736.4 MiB -1028.5 MiB
 83 736.4 MiB -5789.3 MiB
 84 736.4 MiB -4799.5 MiB
 85 
 86 736.4 MiB -4796.0 MiB
 87 
 88 
 89 736.4 MiB -4789.0 MiB
 90 
 91 
 92 
 93 
 94 736.4 MiB -4790.4 MiB
 95 599.3 MiB 6.2 MiB
 96 599.3 MiB 85.1 MiB
 97 599.3 MiB 48.6 MiB
 98 
 99 599.3 MiB 0.0 MiB
 100 599.3 MiB 42.1 MiB
 101 599.3 MiB 0.0 MiB
 102 599.3 MiB 2.3 MiB
 103 
 104 736.4 MiB -4782.2 MiB
 105 736.4 MiB -4767.1 MiB
 106 736.4 MiB -4795.4 MiB
 107 736.4 MiB -4797.3 MiB
 108 
 109 736.4 MiB -4605.9 MiB
 110 
 111 
 112 
 113 736.4 MiB 0.0 MiB
 114 753.1 MiB -807.8 MiB
 115 753.1 MiB -22614.2 MiB
 116 753.1 MiB -21805.3 MiB
 117 753.1 MiB -21806.2 MiB
 118 753.1 MiB -21806.2 MiB
 119 753.1 MiB -12706.5 MiB
 120 
 121 753.1 MiB -12691.2 MiB
 122 753.1 MiB -12707.0 MiB
 123 
 124 753.1 MiB -9099.8 MiB
 125 753.1 MiB -807.8 MiB
 126 753.1 MiB -807.8 MiB
 127 
 128 753.1 MiB -807.8 MiB
 129 753.1 MiB -807.8 MiB
 130 753.1 MiB -807.8 MiB
 131 753.1 MiB -807.8 MiB
 132 
 133 753.1 MiB -807.8 MiB
 134 
 135 753.1 MiB -807.8 MiB
 136 753.1 MiB -807.8 MiB
 137 
 138 
 139 753.1 MiB -807.7 MiB
 140 753.1 MiB -807.7 MiB
 141 
 142 753.1 MiB -807.8 MiB
 143 
 144 752.9 MiB -0.2 MiB
 145 
 146 
 147 
 148 
 149 
 150 
 151 
 152 752.9 MiB 0.0 MiB

[alowayed]

Similar issue here. Totals don't add up and the increments are large and negative.

Line # Mem usage Increment Line Contents
================================================
 5 23.891 MiB 23.891 MiB
 6
 7
 8 23.891 MiB 0.000 MiB
 9 23.891 MiB 0.000 MiB
 10
 11 24.137 MiB 0.246 MiB
 12
 13
 14 112.980 MiB 88.844 MiB
 15
 16 112.980 MiB 0.000 MiB
 17
 18 136.840 MiB -2654.883 MiB
 19
 20 136.840 MiB -2676.617 MiB

[fabianp]

Thanks @nathantsoi and @alowayed for the report. This bug doesn't seem to appear in the standard "test" suite (i.e., typing make test from the root directory). Could you paste a minimal reproducing example for this?

[karan10111]

Any progress on this?

[astrojuanlu]

@karan10111 we would need some help in having a short, self-contained, correct example to reproduce this. Do you have a short snippet that we can use for that?

[alowayed]

@fabianp I'll need some time to reduce the actual code. This appeared while an XML tree was being walked through Hadoop.

[karan10111]

So, most of the times you will see this in loops.
train, test = next(islice(cv.split(X, y), 2, 3))

cv is from sklearn.

[fabianp]

Thanks for the info. That makes sense. When a line is executed multiple times, only the maximum is displayed, which might lead to some confusing information being displayed. PRs or ideas on how to handle this better are welcome.

[astrojuanlu]

Unless we find other cases where this happens, I guess this is a duplicate of #9.

[astrojuanlu]

@fabianp proposed a fix for this in #202. @karan10111 @alowayed @nathantsoi could you please try pip install https://github.com/fabianp/memory_profiler/archive/fix_195.zip and rerun your examples?