{"id":769,"date":"2006-12-29T20:01:24","date_gmt":"2006-12-29T20:01:24","guid":{"rendered":"http:\/\/software.sil.org\/wesay\/?p=769"},"modified":"2017-04-26T23:20:39","modified_gmt":"2017-04-26T23:20:39","slug":"using-python-to-read-a-fieldworks-lexicon","status":"publish","type":"post","link":"https:\/\/software.sil.org\/wesay\/using-python-to-read-a-fieldworks-lexicon\/","title":{"rendered":"Using Python To Read a FieldWorks Lexicon"},"content":{"rendered":"<p>Well, it\u2019s Christmas break for me, which  usually means my head gets hungry to learn something new.  Alas, that hasn\u2019t happened yet, but I did try something new.<\/p>\n<p>I got into <a href=\"http:\/\/pythonnet.sourceforge.net\/\" title=\"\" class=\"external\" target=\"_blank\" rel=\"noopener\">Python for .Net<\/a> and used it to write a trivial <a href=\"..\/..\/fieldworks\" title=\"\" class=\"external\">FieldWorks Language Explorer (FLEx)<\/a> to Standard Format export sample.<\/p>\n<h2>The Vision<\/h2>\n<p>There is a tendency to view FieldWorks as a \u201cblack box\u201d; with SFM\/Toolbox, the data may be a complete inconsistent mess, but hey, it\u2019s just text, and you can \u201cget at it\u201d.  And while ordinary users have much more ability to munge their data using FLEx\u2019s <strong>Bulk Edit<\/strong> tool, there will always be \u201cjust one more feature\u201d needed by one or two people.  I personally don\u2019t believe helping people write SQL stored procedures is the answer.<\/p>\n<p>Rather, since Python is gaining popularity in field linguistic computing, many people would like to see FLEx provide various levels of support for it.  These would include:<\/p>\n<p>A. Write your own python which talks to the FieldWorks Database.<br \/>\nB. Paste in a piece of python, which someone emailed you, into FLEx to do a fix, search, transform, etc. on your data.<br \/>\nC. Setup smart fields which provide default values by running a bit of python script whenever the record changes.  These could then be overridden if the user types in the field.  For example this would work for transductions into other scripts, or sort keys.<\/p>\n<h2>Trying it Out<\/h2>\n<p>I decided to start with the first scenario, running a script external (not embedded) to FLEx to do a trivial export.<\/p>\n<div class=\"announcement\"><p class=\"bold\">UPDATE:<\/p><p>Ken Zook wrote up a bunch on doing this with <a href=\"http:\/\/www.ironpython.com\/\" class=\"external\" target=\"_blank\" rel=\"noopener\">Iron Python<\/a>.<\/p><\/div>\n<p>Python for .Net, unlike Microsoft\u2019s <a href=\"http:\/\/www.ironpython.com\/\" title=\"\" class=\"external\" target=\"_blank\" rel=\"noopener\">Iron Python<\/a>, runs under normal Python.  This, I think, will make it more accessible to field workers wanting to use their Python skills to import, export, or modify FieldWorks databases. They don\u2019t have to purchase and learn Visual Studio or loose access to some libraries.<\/p>\n<p>I got the latest version out of subversion (the released version is quite old), applied the bizarre change mentioned in the VS_Readme, battled a bit with the supplied nant script, and compiled it (i.e., I think there\u2019s room for us to help this project). The Taiwan earthquake broke our fiber connections out of Asia, but I finally found Python2.4 in Japan, installed that, then put the clr.dll and runtime into my python directory. Finally, I installed ActiveState Komodo, told it to use my FieldWorks directory, and wrote the following (this is may be only my second and longest python script ever, so be gentle):<\/p>\n<pre><code>#python defaults to ascii, so switch the console to unicode\n\nimport sys, codecs\nsys.stdout = codecs.getwriter(\u2018utf-8\u2019)(sys.stdout)\n\n#this is the python.net piece that interfaces us to the .net\n#FieldWorks libraries\nimport clr\nfrom CLR.System.Reflection import Assembly\n\n#FDO (FieldWorks Data Objects) is the\n#Object-Relational-Mapper layer of FieldWorks\nfdo = Assembly.LoadWithPartialName(\u201cFDO\u201d)\nfrom CLR.SIL.FieldWorks.FDO import FdoCache\n\n#open up a language project to work on\ndb = FdoCache.Create(\u201cTestLangProj\u201d)\nlp = db.LanguageProject\nvern = lp.DefaultVernacularWritingSystem\nanalysisWs = lp.DefaultAnalysisWritingSystem\nlexicon = lp.LexicalDatabaseOA\nfor e in lexicon.EntriesOC :\n    print \u201c\\lx \u201d + e.LexemeFormOA.Form.GetAlternative(vern)\n    for sense in e.SensesOS :\n        print \u201c\\ge \u201d + sense.Gloss.GetAlternative(analysisWs)\n        if sense.MorphoSyntaxAnalysisRA &lt;&gt; None:\n            print \u201c\\pos \u201c   + sense.MorphoSyntaxAnalysisRA.InterlinearAbbr\n    print\n<\/code><\/pre>\n<p>Here\u2019s the output: <a href=\"\/wp-content\/uploads\/sites\/35\/2017\/04\/20061229_10554140623.png\" rel=\"lightbox\"><br \/>\n<img decoding=\"async\" src=\"\/wp-content\/uploads\/sites\/35\/2017\/04\/20061229_10554140623.png\" alt=\"Python output\" title=\"Python output\" \/><br \/>\n<\/a><\/p>\n<h2>Problems<\/h2>\n<p>Now, there are some issues. A big one is that you don\u2019t get Visual Studio\u2019s Intellisense to tell you what properties and methods are available in FDO.  Is there a way to provide that info to a python IDE some how?  FDO is generated, so we could generate a file for the IDE. I see Komodo has an XML format for this called <a href=\"http:\/\/aspn.activestate.com\/ASPN\/docs\/Komodo\/3.1\/komodo-doc-codeintel.html#codeintel_build_db\" title=\"\" class=\"external\" target=\"_blank\" rel=\"noopener\">cix<\/a>. Is there is such a format for a popular free python editor?<\/p>\n<p>There are also problems accessing the COM objects that FDO exposes here and there.  For example, if you want access to a multi-lingual or formatted string in FW, you currently can\u2019t<br \/>\nget at it from Python.  FDO is c# code, but these objects are lower than FDO, C++ wrapped for .Net. We don\u2019t yet know if there is something that could be changed about that wrapping; IronPython has the exact same problem.  Anyhow, I\u2019m confident that we could wrap those low-level classes, as needed, to make them accessible to python scripters.<\/p>\n<h2>Questions<\/h2>\n<p>I\u2019d like to play around more with this, and the FLEx team may be able to allocate time to it in the future, if folks show enough interest.  But first, can anyone suggest typical scenarios? What would you envision wanting to do to a FLEx database?  Make sort keys? Do complex selections that can then be used in Bulk Edit?  Generate statistics?  Out of these actual code scenarios, we could then think about a new namespace for FLEx scripters which would have friendly names , appropriate abstractions, and good documentation.<\/p>\n<div class=\"top border\"><a href=\"#top\">top<\/a><\/div>\n","protected":false},"excerpt":{"rendered":"<p>Well, it\u2019s Christmas break for me, which usually means my head gets hungry to learn something new. Alas, that hasn\u2019t happened yet, but I did try something new. I got into Python for .Net and used it to write a trivial FieldWorks Language Explorer (FLEx) to Standard Format export sample. The Vision There is a [&hellip;]<\/p>\n","protected":false},"author":26,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[10],"tags":[],"class_list":["post-769","post","type-post","status-publish","format-standard","hentry","category-news"],"acf":[],"_links":{"self":[{"href":"https:\/\/software.sil.org\/wesay\/wp-json\/wp\/v2\/posts\/769","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/software.sil.org\/wesay\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/software.sil.org\/wesay\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/software.sil.org\/wesay\/wp-json\/wp\/v2\/users\/26"}],"replies":[{"embeddable":true,"href":"https:\/\/software.sil.org\/wesay\/wp-json\/wp\/v2\/comments?post=769"}],"version-history":[{"count":5,"href":"https:\/\/software.sil.org\/wesay\/wp-json\/wp\/v2\/posts\/769\/revisions"}],"predecessor-version":[{"id":774,"href":"https:\/\/software.sil.org\/wesay\/wp-json\/wp\/v2\/posts\/769\/revisions\/774"}],"wp:attachment":[{"href":"https:\/\/software.sil.org\/wesay\/wp-json\/wp\/v2\/media?parent=769"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/software.sil.org\/wesay\/wp-json\/wp\/v2\/categories?post=769"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/software.sil.org\/wesay\/wp-json\/wp\/v2\/tags?post=769"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}