$Id: README,v 1.3 2007-01-03 00:44:28 matju Exp $ PureUnity Copyright 2006 by Mathieu Bouchard This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. See file ./COPYING for further informations on licensing terms. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. +-+-+--+---+-----+--------+-------------+---------------------+ GOALS 1. To provide a unit-test framework, which also provide benchmarking features, all made in Pd for use in Pd. 2. To provide tests for functionality in internals, externals, abstractions, etc., in a modularized way, in a DRY/OAOO fashion, thus abstracting out common features so that many objects share the same test patch for the features that they have in common. +-+-+--+---+-----+--------+-------------+---------------------+ REQUIREMENTS 1. Pd 0.39 (PureMSP or Devel) +-+-+--+---+-----+--------+-------------+---------------------+ TEST PROTOCOL new: create common (reusable) fixtures. inlet 0: bang: run all available tests in that class. individual tests don't have to be available through individual methods but may. If they do, the names of the methods must match those given in the test results. each test should build its own non-reusable fixtures and reinitialize common fixtures, not assuming that the previous tests have left the common fixtures in a normal state. outlet 0: test results. a sequence of lists like: list $passed? $accuracy $elapsed $name1 ... where: $passed? is either 0 for failure or 1 for success $accuracy is a float proportional to relative error on math (if not applicable, use 0) $elapsed is a nonnegative float, the time elapsed in milliseconds or it is any negative float meaning the time hasn't been measured. $name1 and the rest are symbols and/or floats identifying the test for example: list 1 0 -1 commutative f + * Which means that the 1st test about commutativity passed ($2=1) because it was perfectly accurate ($3==0) and that we didn't measure the time ($4=-). +-+-+--+---+-----+--------+-------------+---------------------+ SEVERITIES (in decreasing order) * crash: Segmentation Fault, Bus Error, Illegal Instruction, Infinite Loop, etc. You can't deal with those errors at the level of the tests. Maybe there should be a way to tell a test object to skip certain tests, by name, in order to be able to perform as many tests as possible while waiting for a fix. It could become possible to rescue from some of those crashes if Pd supported exceptions (stack-unwinding). * corruption: this may cause future crashes and failures on innocent objects/features. I have no solution for this except to be careful. * post(),error(),pd_error(): Gets printed in the console. The problem is that those can't be handled by the test objects, so someone has to read them and interpret them. Also they prevent test objects to ensure that error conditions produce error messages. * pd_error2(): I wish this would exist. It would be sort of like pd_error() but it would produce a pd message instead, whose selector would be an error code, designed to be both localizable and [route]able. By default, that message would be sent to the console, but there would be an internal class designed to catch those messages. (If stack-unwinding were possible, it would be disabled by default on pd_error2 and could be enabled explicitly by-selector). * failure: a test object reports a problem through outlet 0. * dropout: a failure in realtimeness... difficult for an object to detect. * inaccuracy: a test more or less succeeds but the test detected that the epsilon sucks. +-+-+--+---+-----+--------+-------------+---------------------+ PROTOCOL FOR [error] new: optional argument which would either be a float (e.g. the $0 of the enclosing abstraction) or a pointer. inlet 0: set $scapegoat: replaces the originator of the message by $scapegoat, which can be a float or a pointer error $1 ...: causes its arguments to be concatenated, space-separated (may include floats), and then sent through pd_error using the appropriate originator (scapegoat). list $1 ...: for future use. would use pd_error2() (see README or previous mail). $1 has to be a symbol. +-+-+--+---+-----+--------+-------------+---------------------+ ACCURACY AND ERROR (in math-related unit tests) The "absolute error" between a practical result and the expected value is considered to be the distance between the two value. That is the absolute value of the difference. In the case of positions in 2D, 3D, etc., use the L2-Norm which is a generalized Pythagoras' Theorem: dist^2 = x^2 + y^2 + z^2 + ... A norm is a distance between something and zero. Sometimes you have several practical results for one expected value and must extract a single absolute error out of that. Then you should pick the largest of the individual absolute errors. Sometimes you don't have an expected value, you just have several practical results that you expect to be quite the same. In that case, the absolute error is the "diameter" of those results. The meaning of diameter here is: the largest distance between any two results. If in a single test you must compare 2D errors with 3D errors and 1D errors, etc., you may have to adjust them by dividing the error by the square root of N (N is the number of dimensions). In that case, the resulting value is called a RMS (Root-Mean-Square). The maximum error introduced by just representing a number as a float (instead of an exact value) is at most proportional to the magnitude of the number (e.g. usually 16 million times smaller: about 6 decimals). Also, often we are only interested in relative error, which is absolute error divided by the norm of the expected result, because small absolute errors don't matter much with large results. This is the reason floats exist in the first place. By default, use relative error as the $accuracy in Pd tests. If you don't have an expected result, then compute the relative error as being the absolute error divided by the norm of the average of practical results. In the RMS case of relative error, the norms of expected results should also be adjusted, but both adjustments cancel because they get divided by each other. That means: don't divide by the sqrt(N) at all and you'll get an appropriate result. +-+-+--+---+-----+--------+-------------+---------------------+ TYPE PREFIXES Those have to be prefixes in order to be honored by DOLLSYM: [$1norm] should expand to [fnorm], [lfnorm], [#norm], etc. Those prefixes are necessary in order to achieve polymorphism through abstraction arguments. CURRENT: f float ~ signal FUTURE (from PureData): s symbol p gpointer a anything l list (of whatever) lf list of floats ls list of symbols lp list of pointers FUTURE (from DesireData): t stringpointer L listpointer v varpointer (instance symbol) FUTURE (from GridFlow): # grid (of whatever) #b grid of bytes (uint8) #s grid of shorts (int16) #i grid of ints (int32) #l grid of longs (int64) #f grid of floats (float32) #d grid of doubles (float64) #r grid of rubies (VALUE*) for a type prefix to be considered implemented, it has to have the following class set: metaabstraction for floats for signals for grids [$1.inlet] [inlet] [inlet~] [inlet] [$1.outlet] [outlet] [outlet~] [outlet] [$1.do $2 $3] [$2 $3] [$2~ $3] [# $2 $3] [$1.taa] [t a a] noop [t a a] [$1.swap] [swap] noop TODO [$1.norm] [abs] [env~] [# sq]->[#ravel]->[#fold +]->[#export]->[sqrt] [$1.packunpack3] pack,unpack noop TODO The first two cannot be implemented as abstractions and instead must be defined as aliases in pureunity.c. extra metaabstractions: [$1.rand] [f.rand] [~.rand]TODO [#.rand]TODO +-+-+--+---+-----+--------+-------------+---------------------+ OTHER PROTOCOLS Those four classes are operators that give verify algebraic properties of other operators. The more their outputs are close to zero, the more those other operators are faithful to an algebraic property. (here, supported $types are f and ~) [commutator $type $class] (2 inlets) ab-ba [associator $type $class] (2 inlets) (ab)c-a(bc) [distributor $type $class1 $class2] (3 inlets) a&(b^c)-(a&b^a&c) [invertor $type $class1 $class2] (2 inlets) ab/b-a +-+-+--+---+-----+--------+-------------+---------------------+ TESTS AND RULES For each class, a test file's name is the class name followed by "-test.pd", and a rule file's name is the class name followed by "-rule.pd", in the same way as it is for help files. for a class called $foo, the protocol (aka interface aka rule) $foo is the set of behaviours expected from the $foo class; the class called $foo-rule must repect the $foo protocol as well, plus it should test that the inputs are valid, and if they are, it should test for one or several results and report any errors. (((To report errors and inaccuracies, output them through the properties outlet at the right. If there is no properties outlet in $foo (curently almost nothing in Pd has one), then $foo-rule must have one more outlet than $foo.))) (((Float messages coming out of the properties outlet of $foo-rule report accuracy. Named error messages come out with selector "error" followed by an error-symbol and then its arguments.))) (((In the case of true/false logic, a value of 0 means that a test has passed and a 1 means that a test has failed. Those values represent failure and not success. The reason is so that it matches with accuracy levels, where 0 is perfectly accurate, but any inaccuracy shows up as a relative error fraction. Any finite nonnegative value is allowed for accuracy, because it is expected to be the result of a norm))). (((In standard math, "Discrete Metric" is when there are only two possible distances between objects: together=0 and apart=1))) +-+-+--+---+-----+--------+-------------+---------------------+ RANDOMIZERS (?) +-+-+--+---+-----+--------+-------------+---------------------+ ETC (write me!) If +-test.pd tests [+], it can test for hotness, coldness, it can test that only one result is produced per hot message, that all results are float, that a few example additions work, and that with random inputs it respects commutativity, associativity, invertibility, within appropriate relative-error bounds, etc. However +-test.pd can't test that errormessages aren't printed during the testing. This may be something that we want to check for, and currently the best way to handle it is to search the console for error messages, and if there are any, restart the tests in verbose mode and see where the error happens exactly. [...] Floating-point is the scientific notation for numbers that we all learned on paper in school. Rounding and inaccuracy are two sides of the same coin. They are required when it is stupid to have perfect results, that is, when it would mean too many computations for little gain. However sometimes we want to make sure that our math is accurate enough. Many algorithms are data-recursive: each computation uses previous results. Many of those algorithms have chaotic and/or unstable behaviours, which means that the inaccuracies may skyrocket instead of fading out. +-+-+--+---+-----+--------+-------------+---------------------+ Date: Fri, 13 Jan 2006 04:07:59 +0900 From: Mathieu Bouchard Reply-To: ruby-core@ruby-lang.org To: ruby-core@ruby-lang.org Subject: Re: Design contracts and refactoring (was Re: mathn: ugly warnings) On Fri, 13 Jan 2006, mathew wrote: > *Dean Wampler * writes: > > Let me suggest an XP-style alternative; make thorough unit tests > > required and make sure they "document" - and test! - the design > > "contract". > Unit tests are not an alternative. They are an additional requirement. I find unit-tests to be often decomposable like this. Start with something like this: raise if Blah.new(666) != Blah.new(666) raise if Blah.new(747) != Blah.new(747) raise if Blah.new(242) != Blah.new(242) raise if Blah.new(69) != Blah.new(69) raise if Blah.new(37) != Blah.new(37) then generalize it ("equality is defined based on the arg of .new"): for x in [666,747,242,69,37] do raise if Blah.new(x) != Blah.new(x) end then extract a contract from it: class CheckedBlah < Blah def self.new(x) r = super(x) raise if r != super(x) r end end so now all Blah object creation may be checked throughout actual uses of a program and not just unit tests. The unit test now reduces to: for x in [666,747,242,69,37] do Blah.new(x) end so for many unit tests, all you have to do is just do things and discard the results, and the contract will do the job of checking. _ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801 - http://artengine.ca/matju | Freelance Digital Arts Engineer, Montréal QC Canada +-+-+--+---+-----+--------+-------------+---------------------+ Date: Fri, 13 Jan 2006 05:05:19 +0900 From: Mathieu Bouchard Reply-To: ruby-core@ruby-lang.org To: ruby-core@ruby-lang.org Subject: Re: Design contracts and refactoring (was Re: mathn: ugly warnings) On Fri, 13 Jan 2006, mathew wrote: > For example, consider a simple vector addition routine in a 3D library. > The unit tests might test its behavior with Float and Integer vectors, > since that's why it was written. Here's another way to factor unit-tests that I haven't mentioned in the last mail. suppose you test for + using: class IntegerTest def test; 2+2==4 or raise; end end class FloatTest def test; 2.0+2.0==4.0 or raise; end end class RationalTest def test; Rational(2,1)+Rational(2,1)==Rational(4,1) or raise; end end you can refactor those tests like this: class NumericTest def initialize nt; @nt; end def make x; raise "abstract class" end def test; make(2)+make(2)==make(4) or raise; end end class IntegerTest; def make x; Integer(x) end end class FloatTest; def make x; Float(x) end end class RationalTest; def make x; Rational(x,1) end end > However, to do that you need to know whether the feature of supporting > (say) Complex vectors or BigDecimal vectors is intended or not. The unit > tests won't tell you this. [...] > > One limitation of documentation is that it has no enforcement power, > > so you have to write tests anyway to test conformance. > Unit tests have no enforcement power either, because you can just change the > test. Indeed, I've already had to do this once when it turned out that the > unit test was wrong. (In net/ftp.) That was a pretty bad case of strawman argument. Dean was assuming that your documentation was not executable when you had quite clearly stated that it was the contracts that acted as documentation! [...] +-+-+--+---+-----+--------+-------------+---------------------+ Date: Fri, 13 Jan 2006 07:36:36 +0900 From: Mathieu Bouchard Reply-To: ruby-core@ruby-lang.org To: ruby-core@ruby-lang.org Subject: Re: Design contracts and refactoring (was Re: mathn: ugly warnings) On Fri, 13 Jan 2006, mathew wrote: > > The XP view is > > that you should eliminate the redundancy. > Except it's not redundancy. > Unit tests define a set of functionality that is required. Documentation tells > you the functionality that is supported, which is generally a superset of the > functionality required by the unit tests. Let's follow the argument of both of you to the end. 1. Unit-tests often match inputs with outputs on a case-by-case basis. 2. Redundancy should be eliminated. (1) suggests that there is a shorter way to express the unit-tests. Suppose you are able to find a formula for generating output-validators from inputs. Then that formula is a postcondition of a contract, and the explicit output-validators of the unit-tests are redundant. (2) because part of the unit-tests are redundant, part of the unit-tests should be eliminated. This causes the postconditions to become an essential part of unit-testing. Unit-tests vs contracts is a false debate. _ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801 - http://artengine.ca/matju | Freelance Digital Arts Engineer, Montréal QC Canada +-+-+--+---+-----+--------+-------------+---------------------+ Date: Fri, 13 Jan 2006 17:19:41 +0900 From: Mathieu Bouchard Reply-To: ruby-core@ruby-lang.org To: ruby-core@ruby-lang.org Subject: Re: Design contracts and refactoring (was Re: mathn: ugly warnings) [...] In order to entrench the tests-as-documentation habit firmly in the Ruby community, we need a catchy acronym. Like RTFUT = Read the Fabulous Unit Tests! +-+-+--+---+-----+--------+-------------+---------------------+ http://lists.puredata.info/pipermail/pd-dev/2006-01/005920.html Date: Fri, 20 Jan 2006 23:52:22 -0500 (EST) From: Mathieu Bouchard To: pd-dev Subject: macros and such (was: pd-lib, SIMD) [...] I think that the Pd source doesn't use nearly enough macros or other code-reducing tricks. The reduction of code isn't so much about making things use less RAM: the RAM excuse is quickly evaporating as even the tiniest computers come with plenty of RAM and even the faster kinds of RAM come in ever more copious amounts (big caches). The reduction of code is programmer-oriented. I'm not talking about length of identifiers here (this is a separate issue). Every line of code should do something interesting by itself. Code should read like a good story and not like a car. Ever tried to read a car? It's boring. The same damn piston copy-pasted 12 times. The reduction of code is also documentation-oriented. Once the programmer has been contaminated with the wisdom required to make small code or understand small code, then why wouldn't the programmer explain it to his students in higher-level terms instead of chanting 12 times the same piston as if it were a marathon of Hail-Marys ? This is why Pd needs a taxonomy of object classes. If I don't get that taxonomy in Pd itself nor in its help files, at least I'll have it in its unit tests. Once and only once. Once and only once. Once and only once. Three strikes and you refactor. for x in [1,2,3] say: Once and only once http://c2.com/cgi/wiki/?ThreeStrikesAndYouRefactor BTW I'm not talking about only inheritance of implementations. The most important thing to me is inheritance of expectations, so that if I name 100 classes that obey the rule "Operator2", then you have just learned something common about 100 classes. Operator2 means right-inlet is cold, left-inlet is hot, there is a "set" method for using left-inlet as cold, there is a "bang" for explicitly activating the main computation. The main computation only produces one message. That's what "Operator2" means in my taxonomy, and it's that much that hasn't to be stated explicitly in each help patch. Help patches can be abstractions to be used to by other help patches. Just put a [operator2-help] object in your help patch to indicate that the currently documented class obeys the standard operator2 rules. Who's against it? +-+-+--+---+-----+--------+-------------+---------------------+ http://lists.puredata.info/pipermail/pd-list/2006-02/035169.html Date: Sat Feb 4 21:22:29 CET 2006 From: Mathieu Bouchard To: pd-list * Previous message: [PD] dealing with arguments and inlets * Next message: [PD] Re: [PD-announce] A new version of FFTease is now available for Pd * Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] On Fri, 3 Feb 2006, Hans-Christoph Steiner wrote: > The way I have been thinking is that the first inlet is the general > inlet, and it can accept many types of messages. Then the second inlet > lines up with the first argument, the third inlet to the second > argument, etc. I agree. Many objects obey the rule that the k'th inlet matches argument $k for several arguments in a row, usually all of them. > I think this is pretty clean and flexible, and I think > it would be nice to have some kind of standard for this. And the best way to make sure people are following a standard is to make it so easy to follow that it's harder to not follow it than to follow it. Of course I don't mean adding hurdles to doing it otherwise, but rather make a shortcut for those who follow the standard. Short of this, people who make abstractions/externals can get a friendly reminder, from someone who cares, that it would be better if they followed the standard. > Obviously, it doesn't work for all objects, but I think it would be good to > standardize on objects it does work for. PureUnity's goal (when I work on it) is to design a taxonomy that separates objects that obey certain properties, from those that don't, because that's a way to reuse tests, but also because certainly it doesn't hurt documentation either, and it's even better if it can influence how abstractions are made. _ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801 - http://artengine.ca/matju | Freelance Digital Arts Engineer, Montréal QC Canada +-+-+--+---+-----+--------+-------------+---------------------+ From matju@artengine.ca on Dec 18, 2006 I thought up some kind of classification of type systems, avoiding to call them strong/weak or static/dynamic because those words are confusing. 1. Typed expressions: each piece of code that can give a value, has a type that can be figured out at compile-time. 2. Typed variables/parameters: declarations allow runtime checks but not compile-time checks. 3. Typed values: variables don't have types, they can contain any value, but every value has a type. 4. Typed uses: values don't have types, a type is a way of using a value. Strictness, in the sense of forbidding things to the user, is not on that scale, it's another aspect. A well-balanced strictness allows one to bypass the system whenever needed, but without being too error-prone. However it's difficult to say what it means to "bypass the system" for all four typing categories at once, or even within one category.