Support Issues for MATLAB
Disclaimer - This report contains a combination of fact,
observation, and opinion. It is our best understanding of the
situation under our particular circumstances, and is not
guaranteed to be correct or to apply to conditions other than
those under which the evaluation was made. It is for the use of
the Fermilab / HEP community only.
Maturity and Completeness
- What is the customer base and what is their experience and
opinion?
For commercial software or non-HEP freeware, one
should get a list of customers and references.
There are a number of references on Mathworks' WWW pages. They claim to have
around 400,000 users worldwide. I have asked for HEP
references (users at SLAC, BNL, LANL, etc.) and am waiting.
- How long has the product been in existence?
What
version is the product at? How many major releases have there been? How often
is there a minor release? Several major releases or regular minor releases with
integrated bug fixes are good signs of a well supported mature product.
Availability of published books on the product are also a sign of maturity as
well an established customer base.
The product has been around for ~15 years. Current release is 5.2.
- How long will the product survive?
Are there any
competing products that are likely to win the market (including freeware). Who
is the product developer and are they well supported financially (graduate
student or full time staff).
There is a freeware version, called Octave. It is unclear whether this will
really overtake MATLAB, though. The fact that the Octave graphics are still
very crude indicates that not a lot of work is being applied to areas we
consider important.
Who supports users
- Who provides consulting support? Commercial, other Lab, CERN,
Fermilab? Are they responsive?
Newsgroups and dejanews may provide
some information on support response (though these tend to be biased). This
is rather subjective and should be treated as such.
There are e-mail and phone hotlines for support questions. A local base of
knowledge could easily bedeveloped to help with FAQ and other straightforward
questions. The newsgroup is busy, with about 800 articles posted in the last
two - three weeks.
Particularly for commercial software,
can any user of the product access the support services or are these limited to
a pre-specified list of local contacts.
Anyone at the site would be able to contact the hotline
Is the use of the product in the community enough that there is a
pool of people/knowledge to draw from for support if needed? HEP use
should be assessed; PAW knowledge in the HEP community is widespread and Root
is growing. A dedicated newsgroup would be a plus.
HEP usage is unknown. The newsgroup is busy, and the q & a in there look
healthy, ranging from beginner-level inquiries to refinements of graphics
output. There seem to be plenty of users.
- Is user training needed and available? What is the
cost?
It is available and expensive (about $500/person/2-day course). From my
experience the product is pretty easy to get started with using the doc.
- Is training required and available for support staff? What is
the cost (time and money)?
For commercial products such as SAS and
IDL, support and user training may be required to optimally use the product,
and that cost should be folded in.
There is such a thing, but it isn't yet priced.
- How much (local) support will be required (is it complicated
and hard to use)?
This and the remaining questions in this section can
be determined by talking to current users or scanning any newsgroups, mailists
or FAQS.
The product seems very easy to use. I would think though that a local working
group or mailing list would be helpful, as we write our data interface modules
and other HEP-specific functions. The volume of licenses that we will
require some attention, as well as coordination with the license server people
(fnalu-admin).
- For commercial or freeware, what kind and quality of user level
support is provided?
The helpwin facility and www-based documentation are outstanding. Response to
technical questions from the hotline has been very good. The books and helpwin
are full of examples. Getting started with the product is very easy.
- Is the software completely and well documented at the user
level?
Yes - see above
- Is a system manager required in order to install and/or
maintain the package? If so, this would be significantly complicate
matters for some remote users who do not have ready access to (or a friendly
relationship with) the system managers of their computer.
Not required, but recommended. Workarounds exist for the case where peons do
the installation. UPS/UPD tailoring would be pretty easy.
Licensing
- What types of licensing are available?
Flexlm, floating seats exist for Un*x and NT (these are separate). However,
any of our collaborators can use licenses off of our server. In addition,
there is another breakdown of the Unix licenses into interactive (programmer's)
and batch licenses.
- What is the cost? For Universities? Lab?
The cost is very high. They are quoting ~$3000 per floating Unix seat per year,
with 20% discounts for large volumes.
Maintenance
Who provides it and how much:
- Who provides maintenance both local and external to the Lab?
What are the fallbacks
(if the maintainer(s) is run over by bus or the
company folds)?
I would envision a pool of local knowledge (mailing lists, module repository)
to supplement the hotline. AS a commercial, it would not be "maintained" by
fixing the code, but by feeding questions/problems back to MathWorks and then
updating the local installations. If the company folds we could go to
Octave.
- Are the maintainers responsive and are bug fixes turned around
in a reasonable amount of time?
We didn't encountered any bugs. The support line was responsive in the case of
our two questions.
- Does the software maintainer need additional training
(beyond that needed by users). If so, is it available and at what
cost.
Probably not - just relaying bugs and fixes.
- What is maintenance/licensing costs for commercial products?
Two ways to go. We can by a four year service agreement, in which case we pay
for 4 years of upgrades/hotline/etc. for 20% of our purchase price. This
includes perpetual licenses. Or, we can renew one year at a time, for 20% of
the current price. In this case the license must be renewed each year.
- How much software is there (line count)? How much needs to be
supported locally (how many people required)? Can/should support be split up
into areas of expertise (e.g. motif/graphics, interpreter, etc.). This
is mainly significant for non-commercial software that will be maintained
locally.
Probably N/A.
- IIn the case of commercial software, is source code available
(in escrow)? This would be required for finding bugs locally or in
case the company folds. This may be an additional cost.
It is not available.
Maintenance Infrastructure
- What kind of build environment is provided. Is it
robust?
This is mostly relevant for non-commercial software that may
need to be co-maintained.
Standard build kits for Unix and Install Shield kits for Win32. Distributed on
CD, we could distribute via ZIP files/upd kits probably.
- Can the package be built AT ALL on new or different
sub-systems?
Root still provides NO makefiles.
Kits exist for all of our platforms.
- Is the source repository accessible so that local support
persons can select which changes to accept and which to reject for local
use?
Root still uses CMZ.
N/A
- Will the software have to be maintained and/or extended
locally and externally? If so, can the software be maintained in a common
repository. If separate repositories, what is the commitment to keep them from
diverging from modifications, extensions and bug fixes.
This
excludes locally maintained extensions which use pre-defined APIs or hooks into
the product, which we will have to maintain ourselves in any case.
I can see us sharing modules and controlling those however we choose. But as
far as changing the distribution, it will not occur.
- Is the software passed through quality assurance software such
as Purify or Insure++ before being put into production?
N/A
- Are there any restrictions that would prevent the product from being
placed in the run II infrastructure (i.e. UPS/UPD)? In particular,
the ability to support more than one version of the product on the system.
No such restrictions.
- Are release notes and change lists provided with releases?
For example, the commercial product IDL comes with "what's new" and
release notes lists.
Yes - release notes are provided.
Maturity and Completeness
- Are there active mailing lists/FAQs/newsgroups for the
product?
How do they reflect on the product? Root has a support list,
the commercial product IDL has FAQs and a newsgroup.
All of this exists. THere is a newsgroup where users share ideas and code. The
Mathworks WWW pages are very complete, with FAQ's and the like.
- Are recent releases extensions/enhancements and not bug
fixes?
Yes. Version 5.2 is primarily an upgrade.
- Are product releases reasonably paced and useful?
Yes. Much work is going into Win32 versions.
Modularity:
- Will the tool/software need to be upgraded
(additions/replacements) to satisfy Run II functional requirements, and how
difficult will this be?
Does the product provide API/hooks
to easily interface locally written extensions? Would existing support
be able to handle this or would manpower need to be added? For example
adding a command line (e.g. python) to Histoscope is thought to be difficult.
Anticipated additions and replacements should be identified.
The only big deficiency I see is possibly with the data management. It is
unclear how this will behave when fed a 2GB ntuple. Some code to help manage
such data may have to be written. The API is excellent and well-documented. CD
(PAT, I guess) would probably be able to handle the support with existing
manpower. This would include getting distribution and licenses set up,
upd'ifying the product, and helping the experimentors get going with their data
interface modules. Also preferable would be management of local module base and
mailing list. I see maintenence of this product requiring less work than a
HEP-ware product.
- IIs the product modular? Is it broken down logically and
physically into reasonably distinct sub-systems? Can some of these sub-systems
be replaced by external packages with the same functionality? For
example, in Root can Linear Algebra or Minimization be done by packages
developed specifically to solve these topics separately from Root? A mini code
review should be performed on the package to determine what would be involved
in replacing such an identified component or sub-system.
I didn't actually do it, but I think such a thing could be done without much
problem. For example, it would seem to be straightforward to replace (actually
override) the plotting with OpenInventor graphics, for example.
- If new functionality needs to be added, is the software
sufficiently modular such that the code changes can be localized?
For
example, it is believed this is not the case for adding STL support to cint. A
mini code review should be performed to assess whether extensive and/or
destabilizing code changes would be required to add the functionality.
N/A
- Is the software sufficiently modular to be such that bug fixes
are localized?
A mini code review focused on a particular section or
component of the software should provide information on this.
N/A
- If a component needs to be replaced in it's entirety, is the
software sufficiently modular such that a new component can be slotted in with
minimal disruption?
For example, Root depends on functionality in cint
other than the interpreter in a fundamental way.
??
- What distinct (external) packages or interfaces are required
to build and/or run the package... Motif (shared libraries), OpenGL, etc. Are
there external software components that are out of the maintainer's
control?
LHC++ depends on numerous commercial packages,
Nirvana/Histoscope on motif. All such packages and interfaces should be
identified.
This package is self-contained.
Portability
- Platform availability: Linux, NT, Solaris, IRIX, HP, DEC-Unix
... ?
If a specific Run II platform isn't supported, what would it
take to get support for it should be determined.
All runii platforms are maintained. We could share licenses between Un*X
platforms.
- Porting of code to new platforms; this applies to
non-commercial software that is currently only supported on selected
systems.
The issue to be raised is the ability of the original
developers to accept changes to be incorporated into the base code so any
porting done here is done once (aside from effects of future OS upgrades) and
does not have to be re-done with each release of the software.
N/A
- How sensitive is the package to minor OS and/or compiler
and/or system header variations?
Root under Linux may be sensitive to
which C-libs are in use, which distribution you are using, which kernel you are
using, which system header patches you have applied (esp., cint), and so on.
PAW is sensitive to OS upgrades. This can be determined by looking at support
history in newsgroups, other support logs, or talking to the user community.
- 64 bit considerations: Does the software run on 64 bit
platforms/OS (alpha/Unix, SGI/Unix, future, e.g. merced)?
Will it be
difficult to port to 64 bit systems (a la COMIS for PAW).
Works on 64-bit platforms.
- Are there Endianship and other heterogeneous environment
considerations?
There don't seem to be
- Is the software product build dependent on a specific compiler
or is it compiler independent? If compiles are not needed for the product, are
there any compiler dependencies present in the API used for locally written
extensions?
In particular, if the software needs to be built with Run
II compilers, it should be verified if it can.
Standards:
- Are standards followed. Compiler standards? Library standards
(e.g. STL, POSIX). Are they fully supported
(e.g. cint and
STL)?
- Are there any support and maintenance standards or procedures?
For example, any control over what goes into releases?
N/A
- Are good coding practices (documentation) followed? Is there
good developer's documentation (how easily can the product be "taken
over"?)
N/A
- Are good "Computer Science" techniques and methods used, for
example in a language interpreter (see below)?
N/A
- Is there a design methodology applied? Are any design tools
used? (such as a code generator).
If so, do we need to have and/or
support these tools?
N/A
Reliability and Security:
- Is there any security maintenance concerns with the product?
None
- Is the product likely to crash and if so, how does it
recover?
What would be the impact? Do system managers need to
intervene?
I wasn't able to crash it on NT. There are reports of a crash on the newsgroup,
with shat sounded like a typical seg fault.
- Any government regulation applied to the product?
Export restrictions?
No government regulations. Foreign collaborators could probably make use of our
licenses, but may cost more.
- Are there any Y2K issues?
Application specific:
- Will an interface/adapter need to be made to fit the tool in
with the rest of the analysis tool framework (e.g. data import/export).
Does the product provide well defined and documented API/hooks for
such an extension?
Yes. All work in MATLAB eminates for double precision arrays. Interfaces will
have to be written from experiments' data models.
- Does the product use a language interpreter. If so does it support the
full language. Is it written following correct computer science techniques and
algorithms? Is it sensitive against language changes, etc.
(examples
are COMIS and cint, which support only subsets of their language). Or,
turning the question around, is the language computational complete? (Then
forget about the language it was trying to emulate and treat it as completely
new language).
- Is the data processing model correct?
For example, in
MATLAB, the model is to read in a file then process all its data - will this
work with Run II sized files.
A problem - some cose will have to be written to handle runII sized files.
- Are there restrictions on the input data (say size or
format)?
For example, in Histoscope the size of ntuples is limited
because it is stored and processed in memory.
- What is the minimal environment to run the
product?
How does performance/capability scale up while the
environment scales up?