Some help for a new syntax file (XYplorer)?

[highend]

Hi,

I'm using XYplorer (http://www.xyplorer.com/) as my file manager and it has a scripting engine.

I followed the tutorial / explanations (viewtopic.php?f=9&t=4124) and tried to make a new wordfile for it from scratch.

Most things are working as expected but I have a few questions / problems.

Variables (beginning with a $ sign) are colored fine, but not if they contain an underscore (the second part (_Cmd) is treated like a string, only the first part is colored correctly):

Code: Select all

global $FFmpeg_Cmd;

"Operators" aren't colored at all (ofc I assigned them a color before), e.g. the double equal sign:

Code: Select all

end(exists($Token) == 2), "You've selected a folder but only file(s) are allowed, aborted!";

"Built-in Variables" aren't colored as well. Is this because in the following example they are treated as a string (because of the double quotes)?

Code: Select all

$InputPath = "<curpath>";

This is my xyplorer.uew so far:

Code: Select all

/L1"XYplorer" EnableMLS Nocase Line Comment = // Line Comment Alt = # Block Comment On = /* Block Comment Off = */ File Extensions = xys
/TGBegin "Functions"
/TGFindStr = "%[ ^t]++^([a-z+]*(*)*;^)"
/TGFindStr = "%*=*^([a-z+]*(*)*;^)"
/TGEnd
/TGBegin "Control Structures"
/TGFindStr = "%*if*(*{"
/TGFindStr = "%*foreach*(*{"
/TGFindStr = "%*while*(*{"
/TGFindStr = "%*else*{"
/TGFindStr = "%*elseif*{"
/TGEnd
/TGBegin "Variables"
/TGFindStr = "%[ ^t]++^(^$[a-z_0-9]+^)[ ^t]++=*;"
/TGEnd
/Colors = 0,32768,32768,0,255,
/Colors Back = 16777215,16777215,16777215,16777215,16777215,
/Colors Auto Back = 1,1,1,1,1,
/Font Style = 0,0,0,0,0,
/Delimiters = ! "   $%&'()*+,-./:;<=>?@[\]^_{|}~
/Indent Strings = "{"
/Unindent Strings = "}"
/Open Brace Strings = "{" "(" "["
/Close Brace Strings = "}" ")" "]"
/Open Fold Strings = "{"
/Close Fold Strings = "}"
/C1"Control Structures" STYLE_KEYWORD Colors = 16744448 Colors Back = 16777215 Colors Auto Back = 1 Font Style = 3
;
break
else
elseif
foreach
if
while
/C2"Variables" STYLE_VARIABLE Colors = 33023 Colors Back = 16777215 Colors Auto Back = 1 Font Style = 0
** $
/C3"Operators" STYLE_OPERATOR Colors = 255 Colors Back = 16777215 Colors Auto Back = 1 Font Style = 0
!
%
&
*
+
-
.
/
:
=
?
@
AND
OR
XOR
^
|
~
/C4"Built-in Variables" STYLE_VARIABLE Colors = 16711935 Colors Back = 16777215 Colors Auto Back = 1 Font Style = 0
<base>
<clipboard>
<crlf>
<curbase>
<curext>
<curfolder>
<curitem>
<curitem_dos>
<curitempath>
<curitemprev>
<curlen>
<curname>
<curpath>
<curpath_dos>
<curpath_s>
<cursize>
<curtab>
<curver>
<date>
<focitem>
<items>
<selitems>
<tab>
<title>
<xy>
<xydata>
<xydrive>
<xyexe>
<xyini>
<xypane>
<xypath>
<xyscripts>
<xyver>
%computer%
%desktop%
%net%
%recycler%
%personal%
%desktopreal%
%personalreal%
%winsysdir%
%commonappdata%
%commondesktop%
/C5"Functions" STYLE_FUNCTION Colors = 16711680 Colors Back = 16777215 Colors Auto Back = 1 Font Style = 0
asc
assert
backupto
beep
box
br
button
catalogload
catalogreport
ceil
chr
compare
confirm
continue
copy
copyas
copyitem
copytext
copyto
datediff
delete
download
echo
end
eval
exists
filetype
filter
floor
focus
folderreport
format
formatbytes
formatdate
formatlist
get
getkey
gettoken
global
goto
hash
hexdump
hextodec
highlight
html
incr
input
inputfile
inputfolder
inputselect
isset
isunicode
listfolder
listpane
load
loadsearch
loadtree
makecoffee
md5
moveto
msg
new
now
open
openwith
perm
property
quote
rand
readfile
readpv
readurl
recase
regexreplace
releaseglobals
rename
renameitem
replace
replacelist
report
resolvepath
rotate
round
run
savesettings
sel
selectitems
self
selfilter
seltab
set
setkey
setting
settingp
sortby
sound
status
step
strlen
strpos
strrepeat
sub
substr
swapnames
sync
tab
tag
text
timestamp
toolbar
unset
unstep
urldecode
urlencode
utf8decode
utf8encode
writefile
writepv

[Mofi]

Remove the underscore character from the line starting with Delimiters = and the syntax highlighting engine will not read $FFmpeg_Cmd; instead of $FFmpeg_Cmd; as you would like it.

When the character / is defined in a word list, the line must start with //. Use therefore

// /
:
=

and the operators highlighting starts working.

Highlighting of built-in variables as defined by you in /C4 can't really work because %, < and > are defined as word delimiting characters. You either remove these 3 characters from the list of word delimiters, or you remove all %, <, > from the words in /C4. For second possibility you can keep the delimiter characters % and < at beginning of the words because UltraEdit supports that the first character of a word definition is a delimiter character. But % and > at end of all words must be removed when keeping these 2 characters as word delimiters (because needed for operator highlighting too).

And to answer your next question already yet: It is not possible to get % and > highlighted as operator OR as part of a built-in variable according to what are the characters left and right of these 2 characters. Hint: Use my macro TestForInvalid from The ultimate syntax highlighting tools to detect such invalid word definitions.

Does XYplorer really support strings spanning over multiple lines?

Otherwise you better use DisableMLS instead of EnableMLS in the first line.

You have in the first line whether Noquote nor String Chars = used and therefore UltraEdit uses internal definition String Chars = "' to highlight everything between 2 double or 2 single quotes as string. It is not possible to apply highlighting of words inside strings. For understanding why this is not possible, read my documentation TestForInvalid.htm in the ZIP file containing also the macro TestForInvalid.

[highend]

Wow, lots of invaluable information!

Thanks, really appreciated (and very good explanations for a UEdit beginner).

I wasn't able to get the variables section to work as expected. All variables with <...> are highlighted correctly (I've removed the "%", "<" and ">" from the delimiter list) but all variables with "%..." or "%...%" aren't highlighted. Noquote is present in the first line. But this has a low priority, I don't use these variables so often in my scripts.

May I ask for help on a special regex to identify variables?

E.g.:

Code: Select all

%[ ^t]++^(^$[a-z_0-9]+^)[ ^t]++=*;

This one detects variables if they are at the beginning of the line (with 0-x spaces / tabs in front of it) but how to detect the latter, if you assign one variable to another on the same line like:

Code: Select all

$HiDriveUserName = $UserNameLowerCase;

I've tried various things, e.g:

Code: Select all

%*=*^(^$[a-z_0-9]+^)*

But wasn't able to find a working regex for it.

I've read the help about regular expressions (legacy) (esp. the UE ones) and it should support e.g. the OR parameter via

Code: Select all

^{ A^}^{B^}

But If I try to use this for e.g. the if, else, elseif, while and foreach with:

Code: Select all

%*^{if^}^{else^}^{elseif^}^{while^}^{foreach^}*(*{

it doesn't work.

Regular expressions will stay a mystery (for me)

Tia,
Highend

[Mofi]

Legacy UltraEdit and Unix regular expression engine support only an OR expression with exactly 2 arguments, so just ^{A^}^{B^} for UE engine, but not ^{A^}^{B^}^{C^}. Just the Perl regexp engine supports a nearly unlimited list of arguments in an OR expression.

Syntax highlighting of %variables% works, but of course only when there are word delimiting characters left and right of those variables.

I'm an expert in UltraEdit regular expressions, but I can help you to find suitable expressions without having an example file containing all variants and knowing exactly what you want to see from the example file in the function list and how it should be displayed. Syntax highlighting wordfile for VHDL is an example of what I can do for you when having an appropriate example file and in another text file what should be displayed in the function list view with which structure.

[highend]

Ok,

I have created one file with three sections. Each section shows a few examples, general comments of what should be achieved and the desired output for the examples.
At the end of the file I put all desired output into one (this is what should be visible in the function list window) if everything is working correct.

If you need anything more (examples, comments, explanations or anything else), drop me a message. I will be away over the weekend (until sunday evening). Don't know if I can check this thread in the meantime ;(

Download it if you like:

examples.zip

(1.56 KiB) Downloaded 161 times

[Mofi]

The attached ZIP file contains your example with a few small modifications on control structures example.

Further it contains an XYplorer.uew with updated function strings, new color group /C6 for environment variables as I called this group and new color group /C7 for braces. /C6 is for highlighting all strings between 2 percentage signs using a marker characters definition. The other words in /C6 are not really necessary for highlighting, but are good for auto-completion and auto-correct feature if you use that features. I have also added some more operators.

The regular expression for variables is quite simple. But please note that all occurrences of all variables are listed in the function list.

The functions are listed as you want it, with the exception of end (getinfo("CountSelected") < 1). There is no semicolon after this function, there is a comma. It is possible to include also a comma as end of a function, but that does not solve the problem here because what about getinfo("CountSelected"). Is this a function inside a function? I played a little with this special "function", but the problem is that when interpreting a comma also as termination of a function definition and allow inner round brackets, if, elseif, foreach and while can be also easily found as functions.

You are the first user who wants loops and conditions displayed in the function list view. I tried to define function strings to get them displayed structured in the function list view using the wordfile XYplorer.txt (file extension renamed later), but as you can see on image ControlStructures.png that does not really work. UltraEdit displays the loops and conditions of inner levels also once again in top level resulting in displaying for example the while loop at level 3 in total 4 times.

If leading spaces/tabs are no problem for XYplorer, I suggest to select a block with bad indentation and use Format - ReIndent Selection and do not include loops and conditions in the function list view to see the structure.

[highend]

Thanks a lot for all the effort that you put into this, Mofi!

Is this a function inside a function?

Yes, this is a function embedded into the brackets of a different function.

but as you can see on image ControlStructures.png that does not really work

Yes, I have to admit that. I'm just a newbie regarding programming and I thought it would be a good step if it's possible to see the control structures and their flow in the functions list window but I think I have to reconsider that ;(

One minor little thing that is not working as expected:

Code: Select all

$FullName = "$FirstName" . "$SecondName";

It will only recognize and display the first variable, $FirstName and $Secondname aren't displayed (at this place) in the function list. The question is: does it make sense to display them here (from a programmer's point of view) because they have been defined (not visible in my example, but normally they are) before.

I have to sleep on it for a night

At last, (though I don't know if it's possible):

Variables that are mentioned in comments (either behind a "//", "#" or multiline comment "/*") should probably excluded or they should be displayed with a "Comment:" or "// " prefix in the list.
E.g.:

Code: Select all

// or if it is the second, third, etc. position in a line. All three variable types should be
// listed - Examples: $variable | <curpath> | %desktop%

Function list:

Code: Select all

Variables
// $variable
// <curpath>
// %desktop%


Alternative (different prefix):

Code: Select all

Variables
Comment: $variable
Comment: <curpath>
Comment: %desktop%


Regards,
Highend

[Mofi]

One minor little thing that is not working as expected:
$FullName = "$FirstName" . "$SecondName";

You are right. That's a behavior of the hierachical function list feature I have not known. It looks like if a top level regular expression string has found a string on a line, it does not search further with the same regular expression on this line. That makes sense for most languages because normally at top function string level, function definition lines should be found, and such lines do never contain 2 function definitions on same line. Normally all regular expression search strings at top level contain as first character % which means start search at beginning of a line. Therefore it does not make sense to run the search again on the same line continuing with next character on the line after found string.

But you want something different. Instead of finding only the definition line of a variable, you want all occurences of all variables listed in the function list view. I have never seen this before. A solution is to run a search on second level for more variables on the line with at least 1 variable found. This can be done with following definition:

/TGBegin "Variables"
/TGFindStr = "^(^{^$[a-z_0-9]+^}^{[%<][a-z_0-9]+[%>]^}^)"
/TGBegin "More Variables"
/TGFindStr = "^(^{^$[a-z_0-9]+^}^{[%<][a-z_0-9]+[%>]^}^)"
/TGFindBStart = "="
/TGFindBEnd = ";"
/TGEnd
/TGEnd

I don't know anything about the XYplorer script language syntax, but it looks like global variables are declared with global '$variable_name' and variables are defined with $variable_name =.

So what about searching only for global variable declarations and variable definitions?

/TGBegin "Variables"
/TGFindStr = "%[ ^t]++global[ ^t]+'^(^$[a-z_0-9]+^)'"
/TGFindStr = "%[ ^t]++^(^$[a-z_0-9]+^)[ ^t]++=[~=]"
/TGEnd

That would result in displaying for your example file only following variables in the function list view:

$Counter
$File
$FlashFXPCmd
$FullName
$HiDriveSourceOrDestinationPath
$RSyncSSHOpts
$UserName

Such a function string definition would make much more sense for me. It has also the advantage that variable definitions respectively declarations inside a line comment are ignored automatically, but not inside a multi-line block comment.

The regular expression searches are always done on entire file contents, not just on blocks not highlighted as comment. What an interpreter of a script or a compiler of a program source code interprets from the contents of a file is beyond the knowledge of the general syntax highlighting and function list feature of UltraEdit. With appropriate regular expressions it is often possible to avoid finding strings in a line comment. But in block comments, especially in multi-line block comments, it is often not possible to avoid finding strings in block comments too. Well, definitions of functions or variables inside a block or line comment are nevertheless function and variable definitions. The fact that the interpreter or compiler does ignore them on parsing the file does not automatically mean that such definitions are suddenly no definitions anymore. They are just definitions of currently no interest. A script or source code file should not contain commented code which will never be of interest again in the near future.

[highend]

Code: Select all

/TGBegin "Variables"
/TGFindStr = "^(^{^$[a-z_0-9]+^}^{[%<][a-z_0-9]+[%>]^}^)"
/TGBegin "More Variables"
/TGFindStr = "^(^{^$[a-z_0-9]+^}^{[%<][a-z_0-9]+[%>]^}^)"
/TGFindBStart = "="
/TGFindBEnd = ";"
/TGEnd
/TGEnd

That's nice but it leads to a bigger problem. If the source file is a bit larger (let's say 20-30kb) it takes up to a minute to populate the function list

but it looks like global variables are declared with global '$variable_name' and variables are defined with $variable_name =.

Discard the ' ' and you're right. So the search term should be:

Code: Select all

/TGBegin "Variables"
/TGFindStr = "%[ ^t]++global[ ^t]+^(^$[a-z_0-9]+^)"
/TGEnd

I'll use the second definitions currently and will observe over the next days if I find any anomalies.

So far I'm happy with the full xyplorer.uew file

Thanks again for your support, Mofi!

Regards,
Highend