aboutsummaryrefslogtreecommitdiff
path: root/externals/gridflow/TODO
blob: c6a1818aef302bd06d242d5805d0b1f595bc2b06 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
/* $Id: TODO 3975 2008-07-04 19:57:32Z matju $ */

0.9.5
[ ] get rid of architecture.html
[ ] support "get" in abstractions
[ ] fix [nbx]<->[hsl] relationship in [#camera]
[ ] bugs roman
[ ] add claude's fixes
[ ] other gf mails in inbox
[ ] #line_to_polygon is buggy
[ ] #draw_polygon init_lines() not sufficiently called
[ ] regular dc1394 options
[ ] irregular dc1394 options: white_shading, white_balance
    trigger_mode? trigger, trigger_delay
[ ] Accumulation of Background Statistics
[x] correction/détection de translations dans l'image (avec fft et #remap_image).
[ ] cvEstimateRigidTransform
[ ] compenser pour les ajustements progressifs ou soudains de luminosite
[x] ajouter detection et correction d'interference par fft
[ ] cvEllipse
[ ] cvApproxPoly pour un seul poly `a la fois
[ ] cvMeanShift (?)
[ ] cvCamShift (?)
       Ça s'utilise presque comme MeanShift, sauf que ça requiert un autre type non-grid.
       Celui-ci peut être représenté comme une grid de 5 éléments pour faire plus facile
       (contrairement à cvHistogram, qui contient une grid ND mais a besoin de quelques fields de plus que ça).
[ ] cvCalcBackProject (?)
[ ] cvCalcHist, cvCreateHist.
[ ] on doit aussi commencer à rajouter des types non-grid. Le premier sera CvHistogram et est requis pour tous les objets dont le nom contient "Hist".
[ ] SnakeImage (what is the concept of 'energy', squared luminosity?)
        J'aurais supposé que oui, mais c'est juste trois mesures reliées à trois forces qui "shapent" la courbe:
	espacement égal des points, réduction des angles, et correspondance avec l'image même (qui doit normalement
	être un edge-detect d'une image plus vieille).
[ ] cvOpticalFlow

whatever version
[ ] [cv.split]
[ ] reestablish 'gzfile' feature in a better way. (gzfile is disabled for now)
[ ] ...
[ ] [fps]: translate into pd: "publish if fps>0.001" (to remove outliers)
[ ] conversion from float to int is inconsistent: when to round and when to trunc?
[ ] reimplement negative indices in [listelement] and [listsublist]
[ ] with pwc, 720x480 crashes
[ ] fix SDL modifiers
[ ] fix quoting in [display]

soon:
 * remove reference.*ml
 * videodev.c should be upgraded to V4L2 asap.

+-------+-------+-------+-------+-------+-------+-------+-------
Common Video Framework :

[ ] a common library for all video IO (cameras, files, web), which is the
    project of Thomas H and Johannes.

[ ] a six-way converter for GEM/PDP/GridFlow. There are already
    converters between GEM and PDP and between GEM and GridFlow but
    there is none between PDP and GridFlow and then some things could
    be easier to install and run in a cross-platform way.

[ ] a way to make GEM/PDP/GF conversions *implicitly* instead of
    having objects for that.

[ ] threads for video?
[ ] GEM geos vs 3DP: how do they compare?
[ ] is there anything else deserving to be unified between the video plugins?
[ ] videojack
[ ] http://gstreamer.freedesktop.org/
[ ] distinguish between C++ types Grid* and GridFlow*
[ ] make [#draw_points] [#draw_rect] [#edit_polygon]
[ ] Salsaman recommends supporting Videojack because of unicap support which means getting DV support for free.
[ ] look at camorama
[ ] OpenCV
[ ] Generic Image Library: http://opensource.adobe.com/gil/html/giltutorial.html
[ ] [for] has a bug in "from".
[ ] #out x11 doesn't accept 0-sized grids
[ ] #print   doesn't accept 0-sized grids
[ ] #inner doesnt support seeds of size>1, and isn't vecop-compatible, and ...
[ ] #inner could optimise permutation matrices?
ClaudiusMaximus matju: [#draw_image] inlet 0 method "op <opname>" is undocumented, but very useful :)
ClaudiusMaximus matju: [unix_time] is maldocumented, outlet 0 is a grid Dim[28](uint8), not a symbol

[ ] update doc
[ ] claude wants http://en.wikipedia.org/wiki/Graham_scan

malloc alignment = 4 mod 8 (en utilisant valgrind)

ClaudiusMaximus matju: feature request:  [#store4], behaves like store but accepts grids 
of floats as coordinates and interpolate using 4-point interpolation, eg, right inlet -> 
(3 3 f # 0 0 0 1 1 1 2 4 8), left inlet -> (1 # 0.5) outputs (0.5 0.5 0.5) 
(or whatever the interpolation would really be), left inlet (2 # 1.5 1.5) outputs (3)

http://oprofile.sourceforge.net/about/
also: qprofile, amd profiler, opreport -cl --demangle=smart

[ ] -ftree-vectorizer-verbose
[ ] try removing or modifying -falign on PPC/OSX
[ ] encoding of quicktime RGBA only works with [#cast b] (is that still true?)
[ ] reading tux.tga is completely broken
[ ] install of libquicktime is very wrong on my system (is it 0.9.8?)
[ ] make example on separable convkernels
[ ] add concepts of hot inlet and conf inlet at the code level
[ ] [display] should use clocks to update tk less often
[ ] [display] leaks in server ?
[ ] deprecation warnings
[ ] Add a video latency patch (in examples)
[ ] Numop: warn on misalignment
[ ] remove GridInlet* ?
[ ] configure chokes if GCC3 doesn't have the C++ plugin
[ ] [#import~]
[ ] [#export~]
[ ] [#expr]
[ ] [#line]? [#pipe]? [#delay]?
[ ] #: option swap
[ ] consider including [mysql] [renamefile] [ls] [shell]
[ ] re-enable int64,float64
[ ] added #matrix_solve, #slice
[ ] Copy-on-Write (COW)
[ ] internals documentation
[ ] internal grid protocol redesigned
[ ] raise limit of in/outlets from 10 to 100 or unlimited.
[ ] look into -ftree-vectorize
[ ] look at Tetra Piccolo capture cards, as well as ImpactVCB (Hauppauge)

[145] "list -1"->[@import] only tells error message once every other time
[144] small leak remaining in x11. may have to do with non-use of XDestroyImage.
[140] is [@convolve] actually cross-convolution?
	that is, convolves with a mirror image of the convolution matrix.
[139] [peephole]
	<mis> matju: if you try to move the GOPed abstraction the peepholes don't move along
	<mis> he :)  better yet, the abstraction's bounding box's lower right corner will not
	move up or to the left of the low right corner of the peephole it contains.
	Instead it will resize itself as you move it around.  
[134] #import with zero size crashes?
[133] numop ** can freeze GF
[117] osx: [#store] crash instead of raising
[107] MMX computations may be done misaligned, which may be bad.
[105] when file not found, mpeg error messages are wrong, they say the filename is "".
[102] osx: error: shmget failed (cannot allocate memory) in X11, for regular pictures
[078] "unsupported codec" bogus errors.
[025] format_grid: write test for other bpv's
[030] FormatGrid: not properly tested (crashes sometimes)
[031] stdout should be redirected if bridge is present
[034] format/mpeg3.c,quicktime.c might cause lockup on exit
[038] some seeds aren't typechecked
[039] jpeg file cat crashes on second file (jpeglib buffering?)
[040] GRID_FINISH triggered while backstore ???
[079] can trap segfaults in codecs of libquicktime.
[090] http://poynton.com/notes/colour_and_gamma/ColorFAQ.html#RTFToC30
[041] what would be a way to make GF work well with Pd's idea of a list?
[045] write unit tests about required functionality
[046] write unit tests about error checking
[047] make system to ensure everything is working before a release.
[048] potential issues between fork() and X11 ?
[054] it's possible to crash by sending data in two inlets of a same object at once

--- make the development process more test-driven:
    * file every feature request and bug report into base/test.rb so
      that gridflow fails those tests THEN make gridflow pass those tests.

===================

Installation
[ ] less compile-time checks and more run-time checks
[ ] should GridFlow be bundled with some of its own dependencies?
[ ] find a good profiler (not gridflow's internal...)
[ ] cross-compile:
	http://www.libsdl.org/extras/win32/cross/
	http://rooster.stanford.edu/~ben/linux/cross.php

Documentation
[ ] make a short feature list as a promo
[ ] add credits
[ ] document how to report bugs, basic troubleshooting
[ ] document our distribution of the archives
[ ] ascii conventions: [foo] vs "foo", etc.
[ ] document internals
[ ] describe indexmap in doc
[ ] document backward compatibility issues
[ ] architecture: class->type/shape->use->role
[ ] heavily hyperlink documentation
[ ] automate more parts of the documentation
[ ] benchmarks (especially: simple video i/o)
[ ] mimic [pdp_help_input] [pdp_help_output]: they are convenient
--- remember to keep doc/help/screenshots/examples up to date

Architecture
[0.9] reentrancy of objects (to allow true recursion)
[0.9] inlet-attrs (attrs associated to inlets)
[ ] eliminate the Dim class, replacing by Grid itself (?).
[ ] pull mode (functional-style)
[ ] bidirectional messages (constraint/relational-style)
[ ] send-reply messages (imperative-style)
[ ] add reflection (#global:classes, #global:methods, ...)
[ ] transmitting grids by diffs?
[ ] dispatch on _#{i}_-named methods (any selector one inlet)
[0.9] generalize the backstore
[ ] Outlet#buffer should harmonise with the lcm of its corresponding Inlet#factor's.
[ ] lazy grids: Grid#operator T* makes sure all the grid is received
	and Grid#range(int start, int n, T bogus) makes sure the
	specified part of the grid is received.
[0.9] accounting: grids messages packets numbers bytes copybytes
[0.9] memory profiling
[ ] add a "query" class for debugging other objects?
    (would be able to extract extra info from each object/class)
[ ] add a GridFlow::LaterException or something
    (which tells to retry later with the same packet because another
    grid reception is not completed yet.) 
[ ] interpret nested lists as n-dimensional.
[0.9] MMX memcpy() ?
[0.9] MMX _runtime_ autodetection

Computation Objects
[ ] #counter (similar to #for)
[ ] #for "step" argument should have default {1...}
[ ] zero option in #convolve (?)
[ ] #conv (simpler #convolve) should allow n-D convolution
[ ] #convolve rewritten as combination of #conv+#border ?
[ ] #store: option zero (?)
[ ] #import~, #export~
[ ] #fold, #scan: option reverse (right-to-left)
[ ] #join: option swap
[ ] #matrix_inverse, #matrix_solve
[ ] #timeline (or rtmetro): sequencing of events from grid
[ ] #replace_if, #random_select, #markov_chain, #sort, #slice, #splice
[ ] #polygon_perimeter, #polygon_area, #draw_polygon smoothly
[ ] fuzzy #store (interpolate)
[ ] write a complete benchmark script
[ ] spatial biquads (ask Tom)
[ ] string operations to compensate for the feebleness of PD in that area.
	this includes [#sprintf].
--- see also APL/J/NArray/PDL/Aplus/Yorick/IPOW reference

Effects
[ ] #mosaic
[ ] #gaussian_blur, #sobel, #pixelize, #despeckle (median)
[ ] #dither, #draw_lines, #draw_points, #fire, #queue
[ ] #chroma_key: i0=RGBpicture i1=RGBcolor i2=tolerance o0=RGBApicture
        with fuzzy flag; (don't call it #chroma_key though)
	maybe #mask...
[ ] turtle (logo style)
[ ] effectv
[ ] create abstractions for usage patterns (eg: like #remap_image)
[ ] see gimp for ideas
[ ] kaleidoscopes
[ ] bilinear interpolation a+(b-a)*x+(c-a)*y+(a-b-c+d)*x*y (#scale_by smoothly?)
[ ] #scale_by/#downscale_by fractional values
[ ] bicubic interpolation
[ ] multilinear interpolation (n-dimensional?)
[ ] #daltonism
[ ] #perlin ?
[ ] #moment could support more than 2 dimensions

#<vektor> told me to:
# RGBtoYUV : @fobjects = ["#inner (3 3 # 66 -38 112 128 -74 -94 25 112 -18)","@ >> 8","@ + (16 128 128)"]
# YUVtoRGB : @fobjects = ["@ - (16 128 128)",#inner (3 3 # 298 298 298 0 -100 516 409 -208 0)","@ >> 8"]

Input/Output
[ ] a #store that is persistent (saved in .pd) ?
[ ] Grid: support storage in all six types
[ ] Grid: bpv 1,2,4 uint
[ ] Grid: text version of the format
[ ] Gimp, GStreamer, ffmpeg, xine, mplayer, PDP, OSC
[ ] get subtitling of tv channels
[ ] add messages "load", "save" as open,bang,close combos. (?)
[ ] x11,sdl,aalib: use buffering to synchronise frames at a constant rate.
[ ] x11: sync with the actual vsync of the monitor
[ ] x11: double buffering using a Pixmap remote-object
[ ] [serial_port]
[ ] support for audio in the mpeg streams
[ ] synchronized audio with the video when using mpeg1 & 2
[ ] full screen mode (without the menu bars or frames)
[ ] support for audio in cameras / videodev ... see field vcaps.audios

Format
[ ] remaining number type support in formats
[ ] fork: pseudoformat for launching format handler as separate process
[ ] put_at/reassign in x11/sdl/quartz
[ ] Generic SHM/MMAP Drivers
[ ] "open grid pipe" and such.
[ ] multi: multi-file (by glob pattern, e.g.: foo*.ppm)
[ ] AVI
[ ] format grid: "headerless until 10" (text files)

User Interface
[ ] color picker
[ ] #edit, grid edition dialog, etc.
[ ] vector slider (n parallel sliders in one, Dim(n))
[ ] 2d slider
[ ] radial slider (angle chooser)
[ ] drag-and-drop palette of object construction
[ ] object construction menu specialized for each class (generated from class reflection/metadata)

Extra/Examples/Etc
[ ] .pd: read; write better
[ ] examples/generique.pd
[ ] add examples from the workshop, etc

<matju> alx:
le tab completion ne donnerait pas seulement les noms possibles pour le
prochain argument dans le objectbox; il donnerait aussi une description de
l'argument (surtout dans le cas où l'argument a beaucoup de valeurs
possibles). je pense que le help de Pd est une bonne idée (le concept hands-on)
mais ça pourrait aller plus loin. j'ai pensé intégrer le manuel de référence de
gridflow directement dans le programme, disponible au runtime, dans l'éventualité
d'un GUI qui saurait en tirer avantage.

<alx1> matju: détecter les formes; détecter les couleurs; support mpeg4; hotspots

ClaudiusMaximus reports that allocating a 197M grid causes a crash (turns out C++
throws exception, can't allocate, signal ABRT)

ClaudiusMaximus says: I want to be able to send output from Pd+Gridflow to a video4linux
loopback device so that other v4l programs can access the stream.
I found the video loopback device here: http://motion.sourceforge.net/vloopback/

<alx2>
iSight on linux : http://episteme.arstechnica.com/6/ubb.x?a=tpc&s=50009562&f=8300945231&m=40110681
needs something called 'coriander'. the person who got it working mentions libdc1394.
http://www.tele.ucl.ac.be/PEOPLE/DOUXCHAMPS/ieee1394/cameras/index.html:
I learn that the iSight is compliant with version 1.30 of IIDC.
the docs for coriander mention that the author needed to create a major and minor device for the camera.
retyping (damn pdf's) -> there are two devices to create: raw1394 and video1394
raw1394 has major 171 and minor 0 while video1394 had major 171 and minor 16
any useful info?
having a camera like the iSight supported could be fantastic for video capture/detection on a laptop.  Not to mention that the bttv cards are getting harder to locate

-mfpmath=sse2

remember to check for volatile VALUEs if GC problems ever occur

Tom Schouten wrote: "
added demo/rawvideo.pf and some support scripts in bin/pf-rawvideo.*
this is an example on how to use external programs for raw video input.
rationale:
mplayer is a very nice piece of software supporting virtually any input
format/codec. since i'm not planning to waste time on integrating more
input codecs in pf, the best alternative is to have mplayer (actually
mencoder) or ffmpeg as a subprocess.
the same will be done for recording later.
"

very old bookmarks:
	http://pdl.perl.org/
	http://www.opendragon.com/Pages/MaxObjects.shtml (not responding)
	gstreamer
	Arts Builder
	http://homepages.inf.ed.ac.uk/rbf/HIPR2/wksheets.htm
	http://freej.dyne.org/

FObject.subclass("rubyarray",2,2) {
  def initialize() @a=[]; @i=0; end
  def _0_get(s=nil)
    case s
    when :size; send_out 1,:size,@a.size
    when nil; _0_get :size
    end
  end
  def _0_clear; @a.clear end
  def _0_float i; @i=i; send_out 0, *@a[@i] if @a[@i]!=nil; end
  def _1_list(*l) @a[@i]=l; end
  def _0_save(filename,format=nil)
    f=File.open(filename.to_s,"w")
    if format then @a.each {|x| f.puts(format.to_s % x) }
    else           @a.each {|x| f.puts(x.join(",")) }
    end
    f.close
  end
  def _0_load(filename)
    f=File.open(filename.to_s,"r")
    @a.clear
    f.each {|x| @a.push x.split(",").map {|y| Float(y) rescue y.intern }}
    f.close
  end
}



FObject.subclass("regsub",3,1) {
  def initialize(from,to) _1_symbol(from); _2_symbol(to) end
  def _0_symbol(s) send_out 0, :symbol, s.to_s.gsub(@from, @to).intern end
  def _1_symbol(from) @from = Regexp.new(from.to_s.gsub(/`/,"\\")) end
  def _2_symbol(to)   @to = to.to_s.gsub(/`/,"\\") end
  #doc:_0_symbol,"a string to transform"
  #doc:_1_symbol,"a regexp pattern to be found inside of the string"
  #doc:_2_symbol,"a replacement for the found pattern"
  #doc_out:_0_symbol,"the transformed string"
}

FObject.subclass("memstat",1,1) {
  def _0_bang
    f = File.open("/proc/#{$$}/stat")
    send_out 0, Float(f.gets.split(" ")[22]) / 1024.0
    f.close
  end
  #doc:_0_bang,"lookup process stats for the currently running pd and figure out how much RAM it uses."
  #doc_out:_0_float,"virtual size of RAM in kilobytes (includes swapped out and shared memory)"
}

FObject.subclass("rubysprintf",2,1) {
  def initialize(*format) _1_list(format) end
  def _0_list(*a) send_out 0, :symbol, (sprintf @format, *a).intern end
  alias _0_float _0_list
  alias _0_symbol _0_list
  def _1_list(*format) @format = format.join(" ") end
  alias _1_symbol _1_list
}

there are bugs with abstractions that use [route]->[list] instead of [route2] because they fail when $1="list".

svn propset svn:mime-type text/pure-data *.pd

Benchmarks on a Pentium M 1733 (with #out x11):
(fps)           0.9.0   0.9.3   0.9.4
waves.pd rain    41.1    49.5   50.5
game_of_life.pd 162     183     188
sand.pd          18.7    20.5   20.1
cellular_1d.pd  115     130.1   130
wave_1d.pd       63.0    81.7   81.7

Benchmarks on a Pentium M 1733 (without window):
(fps)           0.9.0   0.9.3   0.9.4
waves.pd rain   50.1            63.0
game_of_life.pd 397             507
sand.pd         27              30
cellular_1d.pd  475             730
wave_1d.pd      107             173