Problem adding custom fields in WeSay 0.9.28.0 for import to Fieldworks 7.0.5~beta5

I thought I had a system for making fields in WeSay, which would then be automatically imported into FLEx, as described here. But just today, I noted that the inability to configure those fields in FLEx is more serious than I had thought. Looking in the FLEx help, one sees:

You cannot change the location or writing system after the custom field is created.

Since custom fields created first in WeSay then imported into FLEx are created in FLEx on import, there is no way to set these options once imported. Complicating this situation, apparently FLEx isn’t taking that (at least writing system) info from WeSay during import. I noticed this when I was moving some data around, and had plural data in an English language field (and this is not an English dictionary…) I went back to WeSay to check the config there; here is the WeSay config for the plural field, above the plural field display in FLEx:

So even though I told WeSay that I just want [nlj] data in this field, on import FLEx set the field as “all analysis, then all vernacular” (which doesn’t seem to match this screenshot, but the point is that I have five language to choose from when inputing data, when I should have just one).
The reason the data is in the ‘en’ field is my fault — I bulk copied to the plural field without checking which writing system I was copying into. But I made this error because I presumed that there was only one language field (this is data, not analysis!) for it to go to. So it is reasonable to imagine others might do so as well, with a bunch of junk language fields that can’t be easily gotten rid of.

Summary

  1. This isn’t the end of the world. One can always create the fields again in FLEx, with the right options, then move all the data over with bulk edit. Needing to do so just negates the value of creating the custom fields on import, unless you don’t care what languages will be available to that field.
  2. Since WeSay clearly has the correct info, hopefully the FLEx team will see this as a bug, and correct the import to take language choice from WeSay (assuming it can understand that [nlj], in this case, is a vernacular writing system, and the systems for categorizing writing systems in WeSay (which selects on a per system basis) and FLEx (which groups languages in categories of first/all vernacular/analysis only/then the other) can be harmonized and/or made to understand one another.
  3. In the mean time, I would advise against importing custom fields from WeSay. We’ll need to take the extra steps to create the fields in each program, and hopefully get it documented clearly enough that each will see the other’s fields the first time around.

Creating a Custom Field II: in Fieldworks 7.0.5~beta5 for WeSay 0.9.28.0

Today I’m going to walk through creating a custom field in Fieldworks, and see how it looks in LIFT and in WeSay.

Fieldworks’ ‘Custom Fields’ Dialog

Creating custom fields in fieldworks is easy, if you know where to look. I created a Tone field via Tools/Configure/Custom fields:

Clicking there produces the Custom Fields dialog box, where one can set up the new field:

Here I have already added Tone and Plural fields. As far as I can tell, there are pros and cons to this method:

  1. Fields added to every record in the database (though I don’t think they take up space, at least in LIFT, until there is data in the field).
  2. Only one of these can appear in a record. I didn’t even notice this until I tried another kind of field (to come), but this may or may not be important to what you’re doing. If you want a couple tone fields for different environments (syntactic, tonal, or whatever), you would need to make them each here, or use another method (description to come).

This is what they look like in FLEx before they have been filled in (Note that I selected different options for the language of these fields):

These fields from the entry in the above screenshot didn’t show up in the LIFT file, since they were empty, but another took the following form (between lexical-unit and senses):

<field type=”Plural”>
<form lang=”gey”>
<text>baadisi</text>
</form>
</field>

And here it is in WeSay:

I saw it immediately on opening WeSay this time, since I had the field already configured earlier, like this:

Note that “Name in file” and “Name for display” are both “Plural”. This makes it a bit easier on the config, since you don’t have to keep track of a different name for the WeSay user to see as in the LIFT file (which is what you see in FLEx).
In the WeSayConfig file, you see this:

<field>
<className>LexEntry</className>
<dataType>MultiText</dataType>
<displayName>Plural</displayName>
<enabled>True</enabled>
<fieldName>Plural</fieldName>
<multiParagraph>False</multiParagraph>
<spellCheckingEnabled>False</spellCheckingEnabled>
<multiplicity>ZeroOr1</multiplicity>
<optionsListFile></optionsListFile>
<visibility>Visible</visibility>
<writingSystems>
<id>gey</id>
</writingSystems>
</field>

Note the fieldName and diplayName values each as ‘Plural.’
When adding (and therefore and naming) a new field in FLEx, that name would show in the same place as Plural in <field type=”Plural”> (the ‘type’ attribute of the field node) for that field in the LIFT file. That would be what you would need to put in the “Name in file” field of the Configuration Tool/Fields dialog above (or in the fieldName field of the WeSayConfig file), in order to see it in WeSay.
A couple caveats for creating custom fields for collaboration between FLEx and WeSay in this manner:

  1. You can’t use spaces. One of the first custom fields I made in FLEx was “Noun Class of Plural.” When I tried to create the corresponding field in WeSay, I got something like this:

    I recall FLEx being perfectly happy writing the field ‘type’ attribute with spaces into the LIFT file, but there was no way to get such a WeSay field, either through the config tool, or through editing the config file by hand. Not that I could find, anyway; perhaps a developer can contradict me here if there is.
  2. A related point is that when creating the field in FLEx first, one is obligated to then create the field in WeSay, or you won’t see it there (the data should still be preserved, but that’s not the kind of collaboration I’m looking for).

But when creating a custom field in WeSay first (As I described here), FLEx creates the field that you created in WeSay automatically. There was a limitation on the options (relative to creating a custom field in FLEx), but going in that direction removes one configuration step for each custom field. So that would depend on the kind of flexibility you need (I haven’t needed those options, yet).
Probably the first issue where I would want those grayed out options would be for fields with option lists. Even in FLEx, the instructions say to set up the options (or at least the list) first, then the field that references them. When trying to collaborate with such a field in WeSay, that would all need to be done first. But I haven’t figured out yet how to get such a field into WeSay, or if the option list fields from WeSay (e.g., POS and SemDom) can go into FLEx, or if they are incompatible data types. If someone figures that one out, please let us all know; if I get time to work on it, I’ll post here.

Notes for creating fields in FLEx’s ‘Custom Fields’ dialog to be used in WeSay

  1. Don’t use spaces in the field name.
  2. Plan on also creating the custom field name in WeSay, with the FLEx field name in the WeSay Configuration Tool’s “Name in file” field.
  3. Don’t use this method for fields that might need to appear more than once per sense/entry, or else make one for each possible iteration you need.
  4. Use this method if you need broader configuration of FLEx custom fields.

Creating Custom Fields in WeSay 0.9.28.0 for Fieldworks 7.0.5~beta5

I’ve been working with custom fields in FLEx and WeSay enough to feel the need to figure out what is really going on. The goal is to be able to straightforwardly create custom fields in one or the the other that are editable and round-trip-able in the other. To do this, I’m going to look into the interface of each program, and see what impact adding fields has on the LIFT (and config, for WeSay) file. Today I’m making a field in WeSay, and seeing what it looks like there, and then in FLEx.

The WeSay Configuration Tool

The WeSay config tool looks like this (once you click on ‘Fields’ then ‘New Field’):

Once you save and exit, you get a section under the <fields> node in the WeSayConfig file that looks like this:

<field>
<className>LexEntry</className>
<dataType>MultiText</dataType>
<displayName>*newField</displayName>
<enabled>True</enabled>
<fieldName>newField</fieldName>
<multiParagraph>False</multiParagraph>
<spellCheckingEnabled>False</spellCheckingEnabled>
<multiplicity>ZeroOr1</multiplicity>
<optionsListFile></optionsListFile>
<visibility>Visible</visibility>
<writingSystems>
<id>en</id>
<id>fr</id>
<id>hav</id>
</writingSystems>
</field>

Adding Data in WeSay

Returning to WeSay, one can add some bogus info to this field in one of the records:

Closing out WeSay and looking at the LIFT file, we see the following under this entry (between <lexical-unit> and the first <sense>):

<field type=”newField”>
<form lang=”fr”>
<text>BogusNewfield</text>
</form>
</field>

What this Means

Putting this all together, we see that

  1. The ‘Name in file’ from the WeSay Config Tool corresponds to the field/fieldName node in the WeSayConfig file.
  2. Both of the above correspond to the LIFT entry/field ‘type’ attribute (once data is entered):
    ‘Name in file’ = (xyz.WeSayConfig)/configuration/components/viewTemplate/fields/field/fieldName = (xyz.lift)/lift/entry/field/@type
  3. ‘Name for display’ from the WeSay Config Tool is the label the WeSay user sees on the field, which corresponds to the contents of the field/displayName node, i.e., (.WeSayConfig)/configuration/components/viewTemplate/fields/field/displayName
  4. Therefore, the name a WeSay user sees for a field will not necessarily relate to anything in FLEx. This is because the WeSay label is related to the proper LIFT field in the WeSayConfig file (which FLEx doesn’t see), and not in the LIFT file, which is what FLEx imports. So in setting up custom fields, we need to pay attention to what the config tool says for the ‘Name in file’, not the ‘Name for display’ (Note that it is ‘*newField,’ and not ‘newField,’ in the WeSay user interface. The asterisk, which is visible in WeSay, is only present in displayName in the WeSayConfig, not in either of fieldName from the WeSayConfig or field/@type from the LIFT file.)

Importing to FLEx

I was happy to see that the field created in WeSay shows up under FLEx custom fields (after importing the WeSay LIFT file):

Note that Location, Type, and Writing System(s) are all grayed out. There may be some way of modifying these settings in FLEx once they have been set in WeSay, but isn’t obvious at first glance. Here is the field in the lexicon editor:

I had to select ‘Show Hidden Fields’ to be able to see it the first time for some reason. But then I deselected it, and the field remained visible.
Note that the label in FLEx is ‘newField,’ without the asterisk, which comes from the type attribute of the field in the LIFT file. As far as I can see, there is no Distinction between file and display names in FLEx. This is appropriate for at least the following two reasons:

  1. FLEx seems to deal fine with spaces in field names (I’ve had problems with this in WeSay).
  2. FLEx users should be able to handle whatever complexity the field names throw at them. WeSay, on the other hand, needs to control carefully what the user sees, and it’s relationship to the LIFT field in question. For instance, the form in lexical-unit in a lift file is displayed as “Word” by default in WeSay, since people are putting words into it. But when I analyze those words into roots, it is nice to be able to change that field’s display name to “Root” in WeSay, without having to change the underlying LIFT structure. This flexibility of the display name can help keep the WeSay user from getting confused without unnecessarily complicating the database.

Notes for Creating fields in WeSay to be imported to FLEx

  1. Pay attention to ‘Name in file’ in the WeSay Config Tool, since that will be what the field will be called in the LIFT file, and in FLEx (and presumably in other programs that would use LIFT).
  2. You may need to click on ‘Show Hidden Fields’ to see the field in FLEx.
  3. There doesn’t seem to be a way to put fields anywhere than in the ‘Custom Fields’ section of FLEx, so I hope that’s where you want it (if not, stay tuned for the next installment, going the other way).

Wesay instructions

When I started working with WeSay in BALSA, it became clear that the people I’m working with were still going to need either a lot of hand-holding, or some instructions. So I wrote these down, and have massaged them a little (not everything was as clear as it could have been), and hope I have something highly (but not completely) fool-proof. But your mileage may vary. I’m submitting them in case someone finds them useful.

Instructions.pdf
Instructions_fr-FR.pdf

Proposal revisited

I’ve written before about using WeSay to collect language data, and at one point I even wrote up a proposal for features I think it would be nice to have there –specifically targeted at orthography development, but the same tools could be used for spell checking (lining up similar profile words, that is, not coming up with a list of correctly spelled words). Anyway, it doesn’t look like that proposal is going anywhere, so I thought I’d give it another try.

The Basic Problem

  1. I’m working in an increasingly large number of languages (five people representing three languages were in my office most of this morning), and I’m looking to see that trend continue (looking to the 60 unwritten languages in eastern DRC).
  2. I am one person, and can only work on one language at a time (however often the language may change throughout the day).
  3. The best tool we have for sorting and manipulating large amounts of lexical data (FLEx) is great at what it does, but is inaccessible to most of the people I work with (it was made for linguists, after all. :-))
  4. So I am left with doing all the work with each guy in FLEx (see #2), or else finding a tool that can be more easily used by the people in #1.

Getting from Data collection to an Orthography

For a number of tasks (word collection), WeSay does exactly what I want. Since most of these people are touching a computer for the first time, let’s get rid of as much of the complexity and room for error as possible. Do one thing at a time, in a constrained environment. But once I’ve collected a wordlist in WeSay, I still need to

  1. Parse the roots out of the forms collected (I can collect plural forms in WeSay now, but we still need to get the full word forms into a field that should be full word forms (e.g., <citation>), and just roots in the lexical-unit field. In case it isn’t obvious, a lot of the phonological analysis depends on the structure of the root — the first root consonant is more important to our analysis that the first word consonant, so we need to be able to sort/filter on it.
  2. Sort and filter the word forms, so we see just one kind of thing at a time (we don’t want to see if kupaka and kukapa have the same ‘p’, since the difference in word position might cause such a difference that is irrelevant to the phonemic system (English speakers pronounce p’s differently in different environments, but use one letter for them all — did you know?).
  3. Go through each controlled list of words, to see where the current writing system (we use a national language to get us started) is not making enough distinctions, and where it is making too many.
  4. Mark changes on the appropriate words, returning the corrected information to the database.

While FLEx can handle all these tasks fine (however slowly at times), if I’m going to help other people move forward in their own language development, I need to find another tool (or tool set). I have got these guys quite happily working in WeSay (yes, there are kinks, but still we all are often working 3-4 times as fast as if I was typing everything myself, since there are more of us typing), but I need to get ready for these next steps, in a way that allows us to build on this momentum, rather than tell everyone but one team to go home until I have more time.

An Attempt

So I’ve been playing with XForms lately (for a lot of other reasons), and I’ve toyed with the idea of getting at and manipulating the LIFT file by another engine. The idea of being able to write a simple form to control the complexity of the underlying XML, and to manipulate it and save back to XML, was very exciting. I even have one form (for collecting noun class permutation examples) deployed. Then I read about the post describing the death of the Firefox XForms extension, and I thought surely, there must be a better way to do this, and I’m sure someone out there knows what it is. So I’ll spend some time outlining exactly what I’d like to see, and maybe someone will know what to do to make it happen.

A Proposal

I took some screenshots of some xforms I did, displayed in firefox. Here’s the first form, for parsing roots (Havu is the name of the language I’m using to test this, and one of the next who will be looking for this to work):

The important aspects are

  1. The prefix of the original citation form (sg here), which filters through the database, allowing us to work on words that are (probably) just of one noun class at a time.
  2. A number of possible plural prefixes, to control the potential output forms
  3. The originally input word form, with gloss (where present), to clearly identify each word.
  4. A number of buttons which allow a non-linguist to simply push a button to input the plural form and parse the root (probably including a “none of the above,” which would skip processing for this word).
  5. Under the hood processing which would, on a click:
  1. copy the original form into a citation form field in the database (potentially a new node in the lift XML),
  2. remove the prefix from the original form, and
  3. input the plural form into a field for the plural form (again, potentially a new node in the lift XML).

It would probably be better to show the user one word at a time, rather than the list in this screen-shot, but I included it all here, to show how the removal of the prefix, and the application of the new prefix, would need to apply to the form of each entry. Also, it would be nice if the form could be adapted for suffixing languages.
What I had so far for an example trigger (in XForms) would be

<xf:trigger><xf:label><xf:output value=”concat(‘mo’, substring(lexical-unit/form[@lang=’hav’]/text[starts-with(.,
‘aka’)], 4))” /></xf:label>
<xf:action ev:event=”DOMActivate”>
<xf:insert context=”.” origin=”instance(‘init’)/citation” />
<xf:setvalue ref=”./citation/form/text” value=“lexical-unit/form[@lang=’hav’]/text[starts-with(., ‘aka’)]”/>
<xf:setvalue ref=“lexical-unit/form[@lang=’hav’]/text[starts-with(., ‘aka’)]” value=“substring(lexical-unit/form[@lang=’hav’]/text[starts-with(., ‘aka’)], string-length(‘aka’) +1)”/>
<xf:send submission=”Save”/>
</xf:action>
</xf:trigger>

As you can see, the value of the prefix is hard-coded here, since I haven’t been able to get variables to work. also, the setvalue expressions don’t really behave (neither node-creation, not setting the right value for an existing node). It’s hard to tell what is a limitation of XForms, and what is a limitation of the Firefox extension –I tried another XForms renderer, but no luck so far… Needless to say, this is not what I do best, so help, anyone?
The next form I’d like sets ATR values for whole words (usually the harmony around here is fairly strict, so it would help with a lot, but not all, vowel questions):

This form is similar to the above, in that I’m looking for a simple regular expression (or a more complicated on, if possible. :-)) to control the data we’re looking at at once, and a binary choice for which vowel group the word belongs to (showing the new word form on the button). A choice of one or another would set the word form accordingly.
Same caveats above about the list of words on a page, and should probably have a “neither” button for when a word doesn’t obey strict vowel harmony, which would bypass processing for that word.
The trigger I had so far (for the -ATR button, reverse for the +ATR) was

<xf:trigger><xf:label><xf:output value=”translate(lexical-unit/form[@lang=’hav’]/text, ‘aeiou’, ‘aɛɨɔʉ’)”/>(-ATR)</xf:label>
<xf:action ev:event=”DOMActivate”>
<xf:setvalue context=”.” value=”translate(lexical-unit/form[@lang=’hav’]/text, ‘aeiou’, ‘aɛɨɔʉ’)”></xf:setvalue>
<xf:send submission=”Save”/>
</xf:action>
</xf:trigger>

The translation includes a>a, but could be modified depending on what /a/ does in a given language (especially if there are 10 vowels).
The third and final form, where we would spend the bulk of our time, might look like this:

Here we have:

  1. The ubiquitous regex to control the data we’re looking at.
  2. The letter that is being evaluated (we’re only analyzing one written form, be it digraph or not, at a time).
  3. The position to find that letter (something like first or second in the root, would probably be good enough).
  4. The options for replacing it (buttons labeled with the new forms), in case a word in the list uses a sound that doesn’t sound the same as the sound in that position in the rest of the words in the list.

Again, a choice of new word form would write the new form to the database (at which point the word would likely disappear from the list, until the regexp matched the new form of that word). It would be nice if the same replacement could be made in the citation and plural forms, presuming the same sounds and written letters would apply. I’m not sure if it would be necessary to devise two forms, one for consonants, and another for vowels. It would depend (at least) on how the regex worked, since the underlying principle is the same for consonants and vowels — look at one thing in one position at a time, and mark each one as the same or different, compared to the other things on the list.

Specifications

Some things we would need to make this tool useful:

  1. Read and modify LIFT format in a predictable and non-destructive way (making all and only the changes we’re looking for, to the fields we’re looking at, leaving everything else alone). This is the format we’re keeping all our lexical data in, and we need to play nice with a number of other programs that use the same data structure standard.
  2. Use of regular expressions. For the root parsing tool, a simple prefix (or suffix) filter would do, but for the others it would be nice to be able to constrain syllable type, as well as position in the word (i.e., kupapata should not appear on the same list as kupapa, even if we’re looking at first ‘p’ in the root, since the second is a longer root). In FLEx, I use expressions like these (more included below), though a simpler format could do, if that kind of power were not possible. Ideally, these expressions would be put into a config file, and the user would only see the label (I have these all done, and can come up with more if needed).
  3. Cross-platform. We run most of our work on BALSA, so it would need to be able to run on at least Linux, which provides BALSA’s OS.
  4. Simple UI. It is probably not possible to overstate computer illiteracy we are dealing with here. People are eager to learn, and often capable to learn, but the less training we need, and the less room a given task has to screw everything else up, the better (WeSay‘s UI is a great model).
  5. Shareable. Even if not open sourced, any tool we might use here needs to be legally put on every computer we and our colleagues use, and they don’t often have the money or access to internet to buy licenses.
  6. Supportable. As hard as we try to keep the possibility of errors out of our workplace, it happens. Today we had two technological problems, each of which required non-significant amounts of my time. If I weren’t here (or if the problems had been beyond me), the teams would have been stuck. The simpler and more accessible (or absolutely error free!) the technology is under the hood, the more likely someone local will be able to deal with problems that arise in a timely manner.

Anyway, there it is.

Examples of Useful Regular Expressions for Filtering Lexical Data

The following are output by a script, which takes as input the kinds of graphs (e.g., d, t, ng’, and ngy) used in a given language’s writing system. For instance, these expressions do not allow ‘rh’ as a single consonant, but those I did for another language does. Similarly, these are based on ten particular vowel letters, which could also be changed for a given language.

^([mn]{0,1})([[ptjfvmlryh]|[bdgkcsznw][hpby]{0,1}])([aiɨuʉeɛoɔʌ])([́̀̌̂]{0,1})([mn]{0,1})([[ptjfvmlryh]|[bdgkcsznw][hpby]{0,1}])([aiɨuʉeɛoɔʌ])([́̀̌̂]{0,1})$

(all CVCV –short vowels only)

^([mn]{0,1})([[ptjfvmlryh]|[bdgkcsznw][hpby]{0,1}])([aiɨuʉeɛoɔʌ]{1,2})([́̀̌̂]{0,1})([mn]{0,1})([[ptjfvmlryh]|[bdgkcsznw][hpby]{0,1}])([aiɨuʉeɛoɔʌ]{1,2})([́̀̌̂]{0,1})$

(all CVCV –long vowels and dypthongs OK)

^([mn]{0,1})([[ptjfvmlryh]|[bdgkcsznw][hpby]{0,1}])([aiɨuʉeɛoɔʌ]{1,2})([́̀̌̂]{0,1})12([aiɨuʉeɛoɔʌ]{1,2})([́̀̌̂]{0,1})$

(all CVCV with C1=C2 –counting prenasalization)

^([mn]{0,1})([[ptjfvmlryh]|[bdgkcsznw][hpby]{0,1}])([aiɨuʉeɛoɔʌ])([́̀̌̂]{0,1})([mn]{0,1})([[ptjfvmlryh]|[bdgkcsznw][hpby]{0,1}])3([́̀̌̂]{0,1})$

(all CVCV with V1=V2 –no long V’s or dypthongs)

^([mn]{0,1})([[ptjfvmlryh]|[bdgkcsznw][hpby]{0,1}])(a)([mn]{0,1})([[ptjfvmlryh]|[bdgkcsznw][hpby]{0,1}])3([́̀̌̂]{0,1})$

(all CVCV with V1=V2=a)

New Wesay Tasks II –Parsing and Tone

Having found out how to configure and add new tasks in WeSay, I can now make some serious changes to my workflow (read: I can actually have people do some of their own data entry and parsing!)

The following task is a follow-up to the one presented here, where the root and meaning were visible but not editable, helping a non-linguist focus on adding plural forms to his dictionary –based on those visible but not editable fields. In this next task, the plural form forms a basis for parsing the root, so the plural and definition fields are visible but the only editable fields are root (EntryLexicalForm, default English label ‘Word’) and citation form. This allows the non-linguist to copy the form from the root field to the citation form field, and remove affixes from the root.
This task is slightly unideal in that it doesn’t know if a word has been parsed yet, or not. It checks for the presence of a citation field (which is usually only made in this kind of process), but it doesn’t know if, for instance, you have a mono-morphemic word which is pronounceable as is (i.e., standard lexicography practice would be to not have a citation field), and it would think that that word was still “to do.”

Here is my new root parser task:

<task
taskName=”AddMissingInfo”
visible=”true”>
<label>Roots</label>
<longLabel>Parse Roots</longLabel>
<description>Move citation info to Citation form field, leave roots in root field.</description>
<field>citation</field>
<showFields>EntryLexicalForm, citation</showFields>
<readOnly>Plural, definition</readOnly>
<writingSystemsToMatch />
<writingSystemsWhichAreRequired />
</task>

While I was at it, I made another task to help populate the “Tone” field. Too bad I can’t get WeSay and FLEx to agree on a writing system to restrict the characters here, so we’re likely to get a lot of malformed data here. But it should be a place to start. Anyway, here it is:

<task
taskName=”AddMissingInfo”
visible=”true”>
<label>Tone</label>
<longLabel>Add Tone Info</longLabel>
<description>Input tone information for each record.</description>
<field>Tone</field>
<showFields>Tone</showFields>
<readOnly>Plural, citation, definition</readOnly>
<writingSystemsToMatch />
<writingSystemsWhichAreRequired />
</task>

This gives me a new desktop:

And a new parsing task:

And a new Tone input task:

And the nice thing about tasks in WeSay, is that you can enable just the ones relevant to the task at hand, to simplify the UI. Which means that I can put all these in my new default project, and leave them disabled (<task taskName=”AddMissingInfo” visible=”false”>, or unticking them in the config) until we need them.
I’ve already shared my dream, but in the mean time, this gets us a lot closer do doing low-level language development in WeSay.

New Wesay Tasks

I found out today that the tasks in WeSay are more configurable than I had thought. I add a “Plural” field to any language project I work in, but until today we have been stuck inputting that either in FLEx (i.e., by me) or in “Dictionary Browse & Edit,” which is unideal at best.
Looking at the config file (in a text editor — make a copy first, etc.!), there are a number of <task/> nodes, several of which have the taskName attribute of “AddMissingInfo.” This appears to be a generic task (unlike some of the others), that comes with a number of options, and can be used multiple times.
In addition to the labeling, there is a field it checks for (if that field is populated, the record doesn’t show in the task), fields it shows as editable, and fields that are shown, but not editable. I don’t know what the last two do, since they are always empty nodes (in all the tasks in my config, anyway). I found out (by trying) that if you tell it to look for a field that isn’t there, it will crash (so make your new fields, then the tasks to add them). I think if you tell it to display a field that isn’t there, it simply won’t. Here is my task node for the new “Add Plurals” task from my config file:

<task
taskName=”AddMissingInfo”
visible=”true”>
<label>Plurals</label>
<longLabel>Add Plurals</longLabel>
<description>Add plural forms to entries where they are missing.</description>
<field>Plural</field>
<showFields>Plural</showFields>
<readOnly>definition, EntryLexicalForm, citation</readOnly>
<writingSystemsToMatch />
<writingSystemsWhichAreRequired />
</task>
Here is the new desktop:

and Here is the page for the new task:

Getting Fieldworks lexical data into XLingpaper

I’ve written before about using WeSay to collect language data, and Wesay lift files can be fairly easily imported into Fieldworks Language Explorer (FLEx) for analysis. Recently, I’ve been working on getting data from the FLEx lexicon into XLingpaper, to facilitate the writing of reports and papers than can be full of data (which is the way I like them…:-)).
I start with a lexicon (basically just a word list) in flex, that has been parsed for root forms (go through noun class categorization and obligatory morphology with a speaker of the language). Figure out the canonical root syllable profile (e.g., around here, usually CVCV), and look for complimentary distribution and contrast within that type, both (though separately) for nouns and verbs.
I have a script that is putting out regular expressions based on what graphs we expect to use (those in Swahili, plus those we have added to orthographies in the past –since we start with data encoded in the Swahili/Lingala orthography, this covers most of the data that we work with). This script puts out expressions like

^bu([mn]{0,1})([[ptjfvmlryh]|[bdgkcsznw][hpby]{0,1}])([aiɨuʉeɛoɔʌ]{1,2})([́̀̌̂]{0,1})$

which means that the whole word/lexeme form (between ^ and $) is just b, u, some consonant (one of m or n, or not, then a consonant letter that appears alone, or the first and second of a digraph), then some vowel (any of ten basic ones, long or short, plus diacritics, or not). In other notation, It is giving buCV, or canonical structures with root initial [bu]. This data is paired with data from other regular expression filters giving [b] before other vowels to show a complete distribution of [b] before all vowels (presumably…).

The script puts out another expression,

^([mn]{0,1})([[ptjfvmlryh]|[bdkgcsznw][hpby]{0,1}])(a)([mn]{0,1})([[ptjfvmlryh]|[bdkgcsznw][hpby]{0,1}])3([́̀̌̂]{0,1})$

which gives me CaCa, as in the following screenshot:

(The 3 refers to the third set of parentheses, (a), so that changing (a) to (i) gives you CiCi.) The data from these filters gives evidence of the independent identity of a vowel, as opposed to vowels created through harmony rules.
So these regular expressions allow filtering of data in the FLEx lexicon to show just the data that you I need to prove a particular point you’re trying to make (in my case, why just these letters should be in the alphabet). But then, how to get the data out of FLEx, and into a document you’re writing?
FLEx has a number of export options out of the box, but none of them seem designed for outputting words with their glosses, based on a particular filter/sort of the lexicon. In particular, I’m looking for export into a format that can be validated against an XLingpaper DTD, since I use XLingpaper XML for most of my writing, both for archivability and longevity of my data, as well as for cross-compatibility in differing environments (there are also developed stylesheets to make XLingpaper docs into html, pdf, and usually word processor docs, too). The basic XML export of the data on the above sort starts like this:

<?xml version=”1.0″ encoding=”utf-8″?>
<ExportedDictionary>
<LexEntry id=”hvo16380″>
<LexEntry_HeadWord>
<AStr ws=”gey”>
<Run ws=”gey”>bana</Run>
</AStr>
</LexEntry_HeadWord>
<LexEntry_Senses>
<LexSense number=”1″ id=”hvo16382″>
<MoMorphSynAnalysisLink_MLPartOfSpeech>
<AStr ws=”en”>
<Run ws=”en”>num</Run>
</AStr>
</MoMorphSynAnalysisLink_MLPartOfSpeech>
<LexSense_Definition>
<AStr ws=”en”>
<Run ws=”en”>four (4)</Run>
</AStr>
</LexSense_Definition>
<LexSense_Definition>
<AStr ws=”fr”>
<Run ws=”fr”>quatre (4)s</Run>
</AStr>
</LexSense_Definition>
<LexSense_Definition>
<AStr ws=”swh”>
<Run ws=”swh”>nne</Run>
</AStr>
</LexSense_Definition>
<LexSense_Definition>
<AStr ws=”pt”>
<Run ws=”pt”>quatro (4)</Run>
</AStr>
</LexSense_Definition>
<LexSense_Definition>
<AStr ws=”es”>
<Run ws=”es”>cuatro</Run>
</AStr>
</LexSense_Definition>
</LexSense>
</LexEntry_Senses>
</LexEntry>
<LexEntry id=”hvo11542″>
<LexEntry_HeadWord>… and so on…

But this is way more information than I need (I got most of these glosses for free using the CAWL to elicit the data), and in the wrong form. The cool thing about XML is that you can take structured information and put in in another structure/form, to get the form you need. To do this, I needed to look (again) and xsl, the extensible stylesheet language, which had succesfully intimidated me a number of times already. But with a little time, energy, and despration, I got a working stylesheet. And with some help from Andy Black, I made it simpler and more straightforward, so that it looks like XLPMultipleFormGlosses.xsl looks today. Put it, and an xml file describing it, into /usr/share/fieldworks/Language Explorer/Export Templates, and this is now a new export process from within FLEx. To see the power of this stylesheet, the above data is now exported from FLEx as

<?xml version=”1.0″ encoding=”utf-8″?>
<!DOCTYPE xlingpaper PUBLIC “-//XMLmind//DTD XLingPap//EN” “XLingPap.dtd”>
<xlingpaper version=”2.8.0″>
<lingPaper>
<section1 id=”DontCopyThisSection”>
<secTitle>Click the [+] on the left, copy the example that appears below, then paste it into your XLingpaper document wherever an example in allowed.</secTitle>
<example num=”examples-hvo16380″>
<listWord letter=”hvo16380″>
<langData lang=”gey”>bana</langData>
<gloss lang=”en”>four (4)</gloss>
<gloss lang=”fr”>quatre (4)s</gloss>
</listWord>
<listWord letter=”hvo11542″>

which contains just enough header/footer to validate against the XLingpaper DTD (so you don’t get errors opening it in XMLmind), and the word forms and just the glosses I want (English and French, but that is easily customizable in the stylesheet). The example node can be copied and pasted into an existing XLingpaper document, which then can eventually be transformed into other formats, like the pdf from this screenshot:

which I think is a pretty cool thing to be able to do, and an advance for the documentation of languages we are working with.