movie.save just hangs after a while with a large amount of frames

[guidov]

Hi Julius, Thanks for sharing this nice little package btw!

My jupyterlab is running on a docker image on a cloud based service (I think through jupyterhub at the University)
I have about 500 x 1920x1080xdpi100 frames. When I use this command:

mov.save("/home/jovyan/work/movie_frames/")

After a while it just stops saving frames, but I when I try to interupt the kernel , the notebook just hangs and I have to restart the kernel to get control back.

I've found a work around where I import frame_save and do the output in batches i.e. this works:

for i in range(300,400):
frame_save(mov.preview(i), i, odir="/home/jovyan/work/movie_frames/", frame_name="frame_",dpi=100)

Anyway, just thought I would mention this , to see if anyone else encounters this in the future, or if its specific to my environment.
I can't diagnose any errors, but you might have a suggestion how to do this

I don't think its a memory or cpu issue as the docker image has 256GB of memory and 64 vCPUs, and the memory doesn't get used up.

[jbusecke]

Thanks for raising the issue! That is indeed concerning.
It might have something to do with the figures not being closed properly. I will definitely check on that.

Could you paste your full code to work around the problem? That would be greatly appreciated.

[guidov]

Hi Julius, I've put the code below from my notebook, but I suspect that it might be something with the way they have configured the jupyterhub service (maybe how the memory is shared between docker images on the server) Not sure. It might be interesting to upload some of my climate data to pangeo and test my notebook there and see if I get similar behavior. Anyhow, I'm not having problems if I save 100 frames at a time. ########################################## # some things are left outside the custom_plotting_func to speed the movie loops up amoc_max = get_amoc_max(docn_fwfp05h_cat["MOC"]) meanp1 = 130 meanp2 = 211 docn_fwfp05h_cat_mean = (docn_fwfp05h_cat["TEMP"][meanp1:meanp2,...].mean(dim="time")).load() def plottemp(ds, fig, tt): #fig = plt.figure(figsize=(16,10)) ########### DATA collect meanp1 = 130 meanp2 = 211 depth = ds["z_t"] lat1=59 lat2=60 #temp_anom_tt = ( ds["TEMP"][tt,...] - (ds["TEMP"][meanp1:meanp2,...].mean(dim="time")) ) temp_anom_tt = ( ds["TEMP"][tt,...] - docn_fwfp05h_cat_mean ) #zpd = ds["PD"] temp_anom_on_atm = regridder(temp_anom_tt) zpd_on_atm = regridder(ds["PD"][tt,...]) temp_anom_on_atm_60N = temp_anom_on_atm.sel(lat=slice(lat1,lat2)).squeeze() zpd_on_atm_60N = zpd_on_atm.sel(lat=slice(lat1,lat2)).squeeze() lons = atmdata["lon"] lats = atmdata['lat'] #print(temp_anom_on_atm_60N) #print(zpd_on_atm_60N) #print(lons) #print(depth) ## velocity #urot, vrot = rotate_uv(ds) #vrot_regrid_tt = regridder(vrot[tt,...]) #vrot_60N = vrot_regrid_tt.sel(lat=slice(lat1,lat2)).squeeze() # do sea ice and MLD on the inset plot mld = regridder(ds.HMXL[tt,:,:]) mld_60N = mld.sel(lat=slice(lat1,lat2)).squeeze() #print(mld_60N) si = regridder(ds.IFRAC[tt,:,:]) mld_cy, lons_cy = add_cyclic_point(mld, lons) si_cy, lons_cy = add_cyclic_point(si, lons) ######################## axisfont=16 axisnumsize=16 lbpx=0.02 lbpy=0.92 lbfs=24 ############################################################## gridon=True # ax2=subplot(211) axisfont=24 tdlevs = np.arange(-5,5,0.2) pdlevs = [1.028, 1.0284] vlevs = [-5,0,5] nlevs=tdlevs.shape[0] lonmin=270 lonmax=360 backgoundcolor='#aaaaaa' tempmap = cmocean.cm.balance # box and line colors bc1='red' bc2='blue' bc3='purple' bclw=3.0 balp=0.8 pdcol='yellow' pdlw=3.0 vcol='white' vlw=2.0 mcol='cyan' mlw=3.0 ############## gs = gridspec.GridSpec(280, 160) # main plot ax2 = plt.subplot(gs[0:100, :-1]) ax2d = plt.subplot(gs[108:170, :-1]) axts = plt.subplot(gs[200:280, :-1]) ####################### upperwater titlestr="60$^\circ$ Latitude Ocean Temperature Anomaly (w.r.t mean D-O cycle)" cp2=ax2.contourf(lons, -0.01*depth, temp_anom_on_atm_60N, tdlevs, cmap=tempmap, extend='both') cp20=ax2.contour(lons, -0.01*depth, temp_anom_on_atm_60N, [0], colors='black', linewidths=1.0, alpha=0.8) cp20l=ax2.contour(lons, -0.01*depth, temp_anom_on_atm_60N, tdlevs, colors='black', linewidths=0.5, alpha=0.8) pdctr = ax2.contour(lons, -0.01*depth, zpd_on_atm_60N, pdlevs, colors=pdcol, linewidths=pdlw, alpha=0.8) #vctr = ax2.contour(lons, -0.01*depth, vrot_60N, vlevs, colors=vcol, linewidths=vlw, alpha=0.8) #mctrd = ax2.plot(lons, -1.0*mld_60N/100., color=mcol, linewidth=mlw, alpha=0.8) ax2.set_xlabel("Time (years)",size=axisfont) ax2.set_title(titlestr,size=24) ax2.set_ylim(-1000,0) adjust_spines(ax2,['left','right','top']) setaxesfsize(ax2,axisfont) ax2.axes.get_xaxis().set_visible(False) if gridon: ax2.yaxis.grid() ax2.xaxis.set_ticks([]) ax2.set_xlim(lonmin,lonmax) ax2.patch.set_color(backgoundcolor) # remove bottom number on left axis ticklabelleft = ax2.get_yticklabels() i=0 for labely in ticklabelleft: if i == 0: labely.set_visible(False) i+=1 ####################### deepwater # ax2d=subplot(212) cp2d=ax2d.contourf(lons, -0.01*depth, temp_anom_on_atm_60N, tdlevs, cmap=tempmap, extend='both') cp2d0=ax2d.contour(lons, -0.01*depth, temp_anom_on_atm_60N, [0], colors='black', linewidths=1.0, alpha=0.8) cp2d0l=ax2d.contour(lons, -0.01*depth, temp_anom_on_atm_60N, tdlevs, colors='black', linewidths=0.5, alpha=0.8) pdctrd = ax2d.contour(lons, -0.01*depth, zpd_on_atm_60N, pdlevs, colors=pdcol, linewidths=pdlw, alpha=0.8) #vctrd = ax2d.contour(lons, -0.01*depth, vrot_60N, vlevs, colors=vcol, linewidths=vlw, alpha=0.8) #mctrd = ax2d.plot(lons, -1.0*mld_60N/100., color=mcol, linewidth=mlw, alpha=0.8) ax2d.set_ylabel("Depth (m)",size=axisfont) ax2d.set_ylim(-4000,-1000) adjust_spines(ax2d,['left','right','bottom']) setaxesfsize(ax2d,axisfont) #ax2d.axes.get_xaxis().set_visible(False) if gridon: ax2d.yaxis.grid() #ax2d.xaxis.set_ticks([]) ax2d.xaxis.tick_bottom() # ax2d.xaxis.tick_top() ax2d.set_xlim(lonmin,lonmax) ax2d.patch.set_color(backgoundcolor) #ax2d.set_xlabel('Latitude ($^\circ$)',size=axisfont) ax2d.set_ylabel('Depth [meters]',size=axisfont) # ax2d.yaxis.set_label_coords(-0.095,1.0) ax2d.yaxis.set_label_coords(-0.06,1.0) ax2d.set_xlabel("Longitude ($^\circ$)",size=axisfont) ax2d.yaxis.set_ticks(np.arange(-4000, 0, 1000)) setaxesfsize(ax2,20) setaxesfsize(ax2d,20) pos = ax2.get_position() l, b, w, h = pos.bounds cbaryoffset=-0.001 ax2.set_position([l,b-cbaryoffset,w,h]) pos = ax2d.get_position() l, b, w, h = pos.bounds cbaryoffset=0.023 cbarxoffset=0.01 ax2d.set_position([l,b+cbaryoffset,w,h]) ctax2d = fig.add_axes([l+w+cbarxoffset, b+cbaryoffset, 0.02, h*2.3]) # setup colorbar axes cb2d=fig.colorbar(cp2,cax=ctax2d, orientation='vertical') # draw colorbar cb2darstrc='Temperature Anomaly ($^\circ$C)' cb2d.set_label(cb2darstrc,fontsize=axisfont,y=0.5) cb2d.ax.tick_params(labelsize=24) # sig0 notation ax2d.text(334, -3500, r'$\sigma_0$ = [28.0, 28.4] kg/m$^{3}$', fontsize=24, color='yellow') # ##############inset a region # Add a second set of axes specifying the fractional coordinates within # the figure with bottom left corner at x=0.55, y=0.58 with width # 0.3 and height 0.25. # Also specify the projection ax_sub = fig.add_axes([0.12, 0.62, 0.25, 0.25], projection=ccrs.NorthPolarStereo(central_longitude=-30)) #ax_sub = plt.axes(projection=ccrs.PlateCarree()) # Add coastlines ax_sub.coastlines(linewidth=2) ax_sub.set_extent([-80, 20, 55, 90], crs=ccrs.PlateCarree()) feat_proj = ccrs.PlateCarree(0.) npts = 100 x, y = np.linspace(0,358, npts), np.linspace(60, 60, npts) ax_sub.plot(x, y, 'red', linewidth = 5, transform=feat_proj) colormap = mpl.cm.jet icemap = mpl.cm.bone contour_levels = np.linspace(0, 1000.0, 21) #ice_levels = np.linspace(10, 100.0, 10) ice_levels = np.linspace(.20, 1.0, 9) cntsplot=ax_sub.contourf(lons_cy, lats, mld_cy/100.,contour_levels, transform=ccrs.PlateCarree(),cmap=colormap,extend='max') iceplot=ax_sub.contourf(lons_cy, lats, si_cy, ice_levels, transform=ccrs.PlateCarree(),cmap=icemap,alpha=1.0) ax_sub.add_feature(feature.OCEAN, zorder=0,facecolor='darkgrey') ax_sub.add_feature(feature.LAND, zorder=0, facecolor='grey', edgecolor='black') pos = ax_sub.get_position() l, b, w, h = pos.bounds cbarxoffset=0.0 cbaryoffset=-0.11 ctax = fig.add_axes([l+cbarxoffset, b+cbaryoffset, w, 0.02]) # setup colorbar axes cb = plt.colorbar(cntsplot, orientation='horizontal',cax=ctax) cbarstrc="Convection Depth (meters)" cb.set_label(cbarstrc,fontsize=18,y=0.5,color='k') cb.ax.tick_params(labelsize=16) #cb.outline.set_edgecolor('white') #cb.outline.set_linewidth(2) #cb.ax.yaxis.set_ticks_position('left') #cb.ax.yaxis.set_label_position('right') #cb.ax.yaxis.set_tick_params(color='white') #plt.setp(plt.getp(cb.ax.axes, 'yticklabels'), color='white',fontsize=lfsize) # set colorbar #cb.ax.invert_yaxis() ibarxoffset=0.0 ibaryoffset=-0.03 itax = fig.add_axes([l+ibarxoffset, b+ibaryoffset, w, 0.02]) # setup colorbar axes ib = plt.colorbar(iceplot, orientation='horizontal',cax=itax) ibarstrc="Sea Ice Fraction" ib.set_label(ibarstrc,fontsize=18,y=0.5,color='k') ib.ax.tick_params(labelsize=16) #ib.outline.set_edgecolor('white') #ib.outline.set_linewidth(2) #ib.ax.xaxis.set_ticks_position('top') #ib.ax.xaxis.set_label_position('top') #ib.ax.yaxis.set_tick_params(color='white') #plt.setp(plt.getp(ib.ax.axes, 'yticklabels'), color='white',fontsize=lfsize) # set colorbar ##################################################### # timeseries plot xmin = 2000 xmax = 6500 amoc_max.plot(ax=axts, linewidth=2.0, alpha=balp, color='b', label="AMOC Max") #axts.plot(ds.time, amoc_max, linewidth=2.0, alpha=balp, color=bc3, label="AMOC Max") msize=200 msalp=0.7 axts.scatter(amoc_max.time[tt].data, amoc_max[tt].data, s=msize, marker='o', color='red', alpha=msalp, edgecolors='k') axts.legend(loc=1, fontsize=20, ncol=1) setaxesfsize(axts, 20) axts.set_xlabel("Time (Years)",size=axisfont) axts.set_ylabel("Atlantic MOC (Sv)",size=axisfont) axts.set_xlim([cftime.DatetimeNoLeap(xmin, 1, 1), cftime.DatetimeNoLeap(xmax, 1, 1)]) axts.set_ylim(0,50) plt.grid() axts.hlines(0, cftime.DatetimeNoLeap(xmin, 1, 1), cftime.DatetimeNoLeap(xmax, 1, 1)) axts.set_title("") mov_temp = Movie(docn_fwfp05h_cat, plottemp, pixelwidth=1920, pixelheight=1080, dpi=100) # full moviesave crashes! mov_temp.save("/home/jovyan/work/movie_frames/")

…

On Wed, Apr 3, 2019 at 3:12 PM Julius Busecke ***@***.***> wrote: Thanks for raising the issue! That is indeed concerning. It might have something to do with the figures not being closed properly. I will definitely check on that. Could you paste your full code to work around the problem? That would be greatly appreciated. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC2IMGW0JUY8TRaCJQgkngcimXYBWnRQks5vdKi7gaJpZM4cUVa3> .

[guidov]

Hi,
The movie.save command seems to work ok on my desktop fine with many frames. I think the problem has to do with running the notebook in the cloud based docker image

[jbusecke]

Ok. Glad that things worked out.

[jbusecke]

I have experienced a similar problem on the GFDL analysis machine. I am not sure what could cause this. But Ill reopen this for now.

[dhruvbalwada]

I have been experiencing the same problem on the pangeo cloud. The kernel crashes after about 30 frames when using the 4-8GB server, and after about 60 frames when using the 8-16GB server. I assume that suggests that it is some sort of memory issue.

For now I have added an option to specify the starting frame number when calling save. This allows me to just restart where things crashed before. However, this is very annoying because I am right now trying to make a movie with over 350 frames, and it will take me over 10 manual restarts.

[dhruvbalwada]

Some more digging around makes me think that the problem is related to #imatplotlib/ipympl/issues/4 matplotlib/ipympl#4 . And might only be occurring when using jupyterlab. As a temporary fix, I ensure that the notebook I am using to make the figures does not call %matplotlib inline magic.

I tried plt.close('all') to close all the open figures, and that option did not work either. In the frame_save function, but even that did not help.

[jbusecke]

Thanks for reporting! This seems like a more substantial problem than I first thought. I currently don't have the bandwidth to investigate in detail, but will def turn my attention to this in the future.