- Added `@inlinedoc` so single use Lua types can be inlined into the
functions docs. E.g.
```lua
--- @class myopts
--- @inlinedoc
---
--- Documentation for some field
--- @field somefield integer
--- @param opts myOpts
function foo(opts)
end
```
Will be rendered as
```
foo(opts)
Parameters:
- {opts} (table) Object with the fields:
- somefield (integer) Documentation
for some field
```
- Marked many classes with with `@nodoc` or `(private)`.
We can eventually introduce these when we want to.
Problem:
The documentation flow (`gen_vimdoc.py`) has several issues:
- it's not very versatile
- depends on doxygen
- doesn't work well with Lua code as it requires an awkward filter script to convert it into pseudo-C.
- The intermediate XML files and filters makes it too much like a rube goldberg machine.
Solution:
Re-implement the flow using Lua, LPEG and treesitter.
- `gen_vimdoc.py` is now replaced with `gen_vimdoc.lua` and replicates a portion of the logic.
- `lua2dox.lua` is gone!
- No more XML files.
- Doxygen is now longer used and instead we now use:
- LPEG for comment parsing (see `scripts/luacats_grammar.lua` and `scripts/cdoc_grammar.lua`).
- LPEG for C parsing (see `scripts/cdoc_parser.lua`)
- Lua patterns for Lua parsing (see `scripts/luacats_parser.lua`).
- Treesitter for Markdown parsing (see `scripts/text_utils.lua`).
- The generated `runtime/doc/*.mpack` files have been removed.
- `scripts/gen_eval_files.lua` now instead uses `scripts/cdoc_parser.lua` directly.
- Text wrapping is implemented in `scripts/text_utils.lua` and appears to produce more consistent results (the main contributer to the diff of this change).
Problem:
The LSP client code is implemented as a complicated closure-class
(class defined in a single function).
Solution:
Move LSP client code to a more conventional Lua class and move to a
separate file.
The benefit of this is that users only pay for what they use. If e.g.
only `vim.lsp.buf_get_clients()` is called then they don't need to load
all modules under `vim.lsp` which could lead to significant startuptime
saving.
Also `vim.lsp.module` is a bit nicer to user compared to
`require("vim.lsp.module")`.
This isn't used for some nested modules such as `filetype` as it breaks
tests with error messages such as "attempt to index field 'detect'".
It's not entirely certain the reason for this, but it is likely it is
due to filetype being precompiled which would imply deferred loading
isn't needed for performance reasons.
Problem:
Currently `deepcopy` hashes every single tables it copies so it can be
reused. For tables of mostly unique items that are non recursive, this
hashing is unnecessarily expensive
Solution:
Port the `noref` argument from Vimscripts `deepcopy()`.
The below benchmark demonstrates the results for two extreme cases of
tables of different sizes. One table that uses the same table lots of
times and one with all unique tables.
| test | `noref=false` (ms) | `noref=true` (ms) |
| -------------------- | ------------------ | ----------------- |
| unique tables (50) | 6.59 | 2.62 |
| shared tables (50) | 3.24 | 6.40 |
| unique tables (2000) | 23381.48 | 2884.53 |
| shared tables (2000) | 3505.54 | 14038.80 |
The results are basically the inverse of each other where `noref` is
much more performance on tables with unique fields, and `not noref` is
more performant on tables that reuse fields.
Problem: luals returns stricter diagnostics with bundled luarc.json
Solution: Improve some function and type annotations:
* use recognized uv.* types
* disable diagnostic for global `vim` in shared.lua
* docs: don't start comment lines with taglink (otherwise LuaLS will interpret it as a type)
* add type alias for lpeg pattern
* fix return annotation for `vim.secure.trust`
* rename local Range object in vim.version (shadows `Range` in vim.treesitter)
* fix some "missing fields" warnings
* add missing required fields for test functions in eval.lua
* rename lsp meta files for consistency
* docs(lua): teach lua2dox how to table
* docs(lua): teach gen_vimdoc.py about local functions
No more need to mark local functions with @private
* docs(lua): mention @nodoc and @meta in dev-lua-doc
* fixup!
Co-authored-by: Justin M. Keyes <justinkz@gmail.com>
---------
Co-authored-by: Justin M. Keyes <justinkz@gmail.com>
Problem:
`vim.split('a:::', ':', {trimempty=true})` trims inner empty items.
Regression from 9c49c10470
Solution:
Set `empty_start=false` when first non-empty item is found.
close#23212
vim.iter wraps a table or iterator function into an `Iter` object with
methods such as `filter`, `map`, and `fold` which can be chained to
produce iterator pipelines that do not create new tables at each step.
feat(lua)!: add stricter vim.tbl_islist(), rename vim.tbl_isarray()
Problem: `vim.tbl_islist` allows gaps in tables with integer keys
("arrays").
Solution: Rename `vim.tbl_islist` to `vim.tbl_isarray`, add new
`vim.tbl.islist` that checks for consecutive integer keys that start
from 1.
* feat(lua): vim.tbl_contains supports general tables and predicates
Problem: `vim.tbl_contains` only works for list-like tables (integer
keys without gaps) and primitive values (in particular, not for nested
tables).
Solution: Rename `vim.tbl_contains` to `vim.list_contains` and add new
`vim.tbl_contains` that works for general tables and optionally allows
`value` to be a predicate function that is checked for every key.
Problem:
semver specifies that digit sequences in a prerelease string should be
compared as numbers, not lexically: https://semver.org/#spec-item-11
> Precedence for two pre-release versions with the same major, minor,
> and patch version MUST be determined by comparing each dot separated
> identifier from left to right until a difference is found as follows:
> 1. Identifiers consisting of only digits are compared numerically.
> 2. Identifiers with letters or hyphens are compared lexically in ASCII sort order.
> 3. Numeric identifiers always have lower precedence than non-numeric identifiers.
> 4. A larger set of pre-release fields has a higher precedence than a smaller set, if all of the preceding identifiers are equal.
Example:
1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-alpha.beta < 1.0.0-beta < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0.
Solution:
cmp_prerel() treats all digit sequences in a prerelease string as
numbers. This doesn't _exactly_ match the spec, which specifies that
only dot-delimited digit sequences should be treated as numbers...
Problem:
- vim.split has more features than vim.gsplit.
- Cannot inspect the "separator" segments of vim.split or vim.gsplit.
Solution:
- Move common implementation from vim.split into vim.gsplit.
- TODO: deprecate vim.split in favor of vim.totable(vim.gsplit())?
- Introduce `keepsep` parameter.
Related: 84f66909e4
Problem:
Build is not reproducible, because generated source files (.c/.h/) are not
deterministic, mostly because Lua pairs() is unordered by design (for security).
https://github.com/LuaJIT/LuaJIT/issues/626#issuecomment-707005671https://www.lua.org/manual/5.1/manual.html#pdf-next
> The order in which the indices are enumerated is not specified [...]
>
>> The hardening of the VM deliberately randomizes string hashes. This in
>> turn randomizes the iteration order of tables with string keys.
Solution:
- Update the code generation scripts to be deterministic.
- That is only a partial solution: the exported function
(funcs_metadata.generated.h) and ui event
(ui_events_metadata.generated.h) metadata have some mpack'ed
tables, which are not serialized deterministically.
- As a workaround, introduce `PRG_GEN_LUA` cmake setting, so you can
inject a modified build of luajit (with LUAJIT_SECURITY_PRN=0)
that preserves table order.
- Longer-term we should change the mpack'ed data structure so it no
longer uses tables keyed by strings.
Closes#20124
Co-Authored-By: dundargoc <gocdundar@gmail.com>
Co-Authored-By: Arnout Engelen <arnout@bzzt.net>
While `return` and `return nil` are for most intents and purposes
identical, there are situations where they're not. For example,
calculating the amount of values via the `select()` function will yield
varying results:
```lua
local function nothing() return end
local function null() return nil end
select('#', nothing()) -- 0
select('#', null()) -- 1
```
`vim.tbl_get` currently returns both nil and no results, which makes it
unreliable to use in certain situations without manually accounting for
these discrepancies.
Problem:
- pesc() returns multiple results, it should return a single result.
- tbl_islist() returns non-boolean in some branches.
- Docstring: @generic must be declared first
Solution:
Constrain docstring annotations.
Fix return types.
Co-authored-by: Justin M. Keyes <justinkz@gmail.com>
- Improve generated HTML by updating parser which includes fixes for
single "'" and single "|":
https://github.com/neovim/tree-sitter-vimdoc/pull/31
- Updated parser also fixes the conceal issue for "help" highlight
queries https://github.com/neovim/tree-sitter-vimdoc/issues/23 by
NOT including whitespace in nodes.
- But this means we need to restore the getws() function which scrapes
leading whitespace from the original input (buffer).
