feat(pack): add built-in plugin manager vim.pack

Problem: No built-in plugin manager

Solution: Add built-in plugin manager

Co-authored-by: Lewis Russell <lewis6991@gmail.com>

runtime/doc/lua.txt

@@ -2525,6 +2525,192 @@ vim.loader.reset({path})                                  *vim.loader.reset()*
       • {path}  (`string?`) path to reset
 
 
+==============================================================================
+Lua module: vim.pack                                                *vim.pack*
+
+WORK IN PROGRESS built-in plugin manager! Early testing of existing features
+is appreciated, but expect breaking changes without notice.
+
+Manages plugins only in a dedicated *vim.pack-directory* (see |packages|):
+`$XDG_DATA_HOME/nvim/site/pack/core/opt`. Plugin's subdirectory name matches
+plugin's name in specification. It is assumed that all plugins in the
+directory are managed exclusively by `vim.pack`.
+
+Uses Git to manage plugins and requires present `git` executable of at least
+version 2.36. Target plugins should be Git repositories with versions as named
+tags following semver convention `v<major>.<minor>.<patch>`.
+
+Example workflows ~
+
+Basic install and management:
+• Add |vim.pack.add()| call(s) to 'init.lua': >lua
+
+    vim.pack.add({
+      -- Install "plugin1" and use default branch (usually `main` or `master`)
+      'https://github.com/user/plugin1',
+
+      -- Same as above, but using a table (allows setting other options)
+      { src = 'https://github.com/user/plugin1' },
+
+      -- Specify plugin's name (here the plugin will be called "plugin2"
+      -- instead of "generic-name")
+      { src = 'https://github.com/user/generic-name', name = 'plugin2' },
+
+      -- Specify version to follow during install and update
+      {
+        src = 'https://github.com/user/plugin3',
+        -- Version constraint, see |vim.version.range()|
+        version = vim.version.range('1.0'),
+      },
+      {
+        src = 'https://github.com/user/plugin4',
+        -- Git branch, tag, or commit hash
+        version = 'main',
+      },
+    })
+
+    -- Plugin's code can be used directly after `add()`
+    plugin1 = require('plugin1')
+<
+• Restart Nvim (for example, with |:restart|). Plugins that were not yet
+  installed will be available on disk in target state after `add()` call.
+• To update all plugins with new changes:
+  • Execute |vim.pack.update()|. This will download updates from source and
+    show confirmation buffer in a separate tabpage.
+  • Review changes. To confirm all updates execute |:write|. To discard
+    updates execute |:quit|.
+
+Switch plugin's version:
+• Update 'init.lua' for plugin to have desired `version`. Let's say, plugin
+  named 'plugin1' has changed to `vim.version.range('*')`.
+• |:restart|. The plugin's actual state on disk is not yet changed.
+• Execute `vim.pack.update({ 'plugin1' })`.
+• Review changes and either confirm or discard them. If discarded, revert any
+  changes in 'init.lua' as well or you will be prompted again next time you
+  run |vim.pack.update()|.
+
+Freeze plugin from being updated:
+• Update 'init.lua' for plugin to have `version` set to current commit hash.
+  You can get it by running `vim.pack.update({ 'plugin-name' })` and yanking
+  the word describing current state (looks like `abc12345`).
+• |:restart|.
+
+Unfreeze plugin to start receiving updates:
+• Update 'init.lua' for plugin to have `version` set to whichever version you
+  want it to be updated.
+• |:restart|.
+
+Remove plugins from disk:
+• Use |vim.pack.del()| with a list of plugin names to remove. Make sure their
+  specs are not included in |vim.pack.add()| call in 'init.lua' or they will
+  be reinstalled.
+
+Available events to hook into ~
+• *PackChangedPre* - before trying to change plugin's state.
+• *PackChanged* - after plugin's state has changed.
+
+Each event populates the following |event-data| fields:
+• `kind` - one of "install" (install on disk), "update" (update existing
+  plugin), "delete" (delete from disk).
+• `spec` - plugin's specification.
+• `path` - full path to plugin's directory.
+
+
+*vim.pack.Spec*
+
+    Fields: ~
+      • {src}       (`string`) URI from which to install and pull updates. Any
+                    format supported by `git clone` is allowed.
+      • {name}?     (`string`) Name of plugin. Will be used as directory name.
+                    Default: `src` repository name.
+      • {version}?  (`string|vim.VersionRange`) Version to use for install and
+                    updates. Can be:
+                    • `nil` (no value, default) to use repository's default
+                      branch (usually `main` or `master`).
+                    • String to use specific branch, tag, or commit hash.
+                    • Output of |vim.version.range()| to install the
+                      greatest/last semver tag inside the version constraint.
+
+
+vim.pack.add({specs}, {opts})                                 *vim.pack.add()*
+    Add plugin to current session
+    • For each specification check that plugin exists on disk in
+      |vim.pack-directory|:
+      • If exists, do nothin in this step.
+      • If doesn't exist, install it by downloading from `src` into `name`
+        subdirectory (via `git clone`) and update state to match `version`
+        (via `git checkout`).
+    • For each plugin execute |:packadd| making them reachable by Nvim.
+
+    Notes:
+    • Installation is done in parallel, but waits for all to finish before
+      continuing next code execution.
+    • If plugin is already present on disk, there are no checks about its
+      present state. The specified `version` can be not the one actually
+      present on disk. Execute |vim.pack.update()| to synchronize.
+    • Adding plugin second and more times during single session does nothing:
+      only the data from the first adding is registered.
+
+    Parameters: ~
+      • {specs}  (`(string|vim.pack.Spec)[]`) List of plugin specifications.
+                 String item is treated as `src`.
+      • {opts}   (`table?`) A table with the following fields:
+                 • {load}? (`boolean`) Load `plugin/` files and `ftdetect/`
+                   scripts. If `false`, works like `:packadd!`. Default
+                   `true`.
+
+vim.pack.del({names})                                         *vim.pack.del()*
+    Remove plugins from disk
+
+    Parameters: ~
+      • {names}  (`string[]`) List of plugin names to remove from disk. Must
+                 be managed by |vim.pack|, not necessarily already added to
+                 current session.
+
+vim.pack.get()                                                *vim.pack.get()*
+    Get data about all plugins managed by |vim.pack|
+
+    Return: ~
+        (`table[]`) A list of objects with the following fields:
+        • {spec} (`vim.pack.SpecResolved`) A |vim.pack.Spec| with defaults
+          made explicit.
+        • {path} (`string`) Plugin's path on disk.
+        • {active} (`boolean`) Whether plugin was added via |vim.pack.add()|
+          to current session.
+
+vim.pack.update({names}, {opts})                           *vim.pack.update()*
+    Update plugins
+    • Download new changes from source.
+    • Infer update info (current/target state, changelog, etc.).
+    • Depending on `force`:
+      • If `false`, show confirmation buffer. It lists data about all set to
+        update plugins. Pending changes starting with `>` will be applied
+        while the ones starting with `<` will be reverted. It has special
+        in-process LSP server attached to provide more interactive features.
+        Currently supported methods:
+        • 'textDocument/documentSymbol' (`gO` via |lsp-defaults| or
+          |vim.lsp.buf.document_symbol()|) - show structure of the buffer.
+        • 'textDocument/hover' (`K` via |lsp-defaults| or
+          |vim.lsp.buf.hover()|) - show more information at cursor. Like
+          details of particular pending change or newer tag.
+        Execute |:write| to confirm update, execute |:quit| to discard the
+        update.
+      • If `true`, make updates right away.
+
+    Notes:
+    • Every actual update is logged in "nvim-pack.log" file inside "log"
+      |stdpath()|.
+
+    Parameters: ~
+      • {names}  (`string[]?`) List of plugin names to update. Must be managed
+                 by |vim.pack|, not necessarily already added to current
+                 session. Default: names of all plugins added to current
+                 session via |vim.pack.add()|.
+      • {opts}   (`table?`) A table with the following fields:
+                 • {force}? (`boolean`) Whether to skip confirmation and make
+                   updates immediately. Default `false`.
+
+
 ==============================================================================
 Lua module: vim.uri                                                  *vim.uri*
 

runtime/doc/news.txt

@@ -202,6 +202,7 @@ LUA
 • |vim.version.range()| output can be converted to human-readable string with |tostring()|.
 • |vim.version.intersect()| computes intersection of two version ranges.
 • |Iter:take()| and |Iter:skip()| now optionally accept predicates.
+• Built-in plugin manager |vim.pack|
 
 OPTIONS
 

runtime/doc/options.txt

@@ -2476,6 +2476,8 @@ A jump table for the options with a short description can be found at |Q_op|.
 		|MenuPopup|,
 		|ModeChanged|,
 		|OptionSet|,
+		|PackChanged|,
+		|PackChangedPre|,
 		|QuickFixCmdPost|,
 		|QuickFixCmdPre|,
 		|QuitPre|,

runtime/example_init.lua

@@ -86,3 +86,8 @@ end, { desc = 'Print the git blame for the current line' })
 -- For example, to add the "nohlsearch" package to automatically turn off search highlighting after
 -- 'updatetime' and when going to insert mode
 vim.cmd('packadd! nohlsearch')
+
+-- [[ Install plugins ]]
+-- Nvim functionality can be extended by installing external plugins.
+-- One way to do it is with a built-in plugin manager. See `:h vim.pack`.
+vim.pack.add({ 'https://github.com/neovim/nvim-lspconfig' })

runtime/ftplugin/nvim-pack.lua

@@ -0,0 +1,47 @@
+local ns = vim.api.nvim_create_namespace('nvim.pack.confirm')
+vim.api.nvim_buf_clear_namespace(0, ns, 0, -1)
+
+local priority = 100
+local hi_range = function(lnum, start_col, end_col, hl, pr)
+  --- @type vim.api.keyset.set_extmark
+  local opts = { end_row = lnum - 1, end_col = end_col, hl_group = hl, priority = pr or priority }
+  vim.api.nvim_buf_set_extmark(0, ns, lnum - 1, start_col, opts)
+end
+
+local header_hl_groups =
+  { Error = 'DiagnosticError', Update = 'DiagnosticWarn', Same = 'DiagnosticHint' }
+local cur_header_hl_group = nil
+
+local lines = vim.api.nvim_buf_get_lines(0, 0, -1, false)
+for i, l in ipairs(lines) do
+  local cur_group = l:match('^# (%S+)')
+  local cur_info = l:match('^Path: +') or l:match('^Source: +') or l:match('^State[^:]*: +')
+  if cur_group ~= nil then
+    --- @cast cur_group string
+    -- Header 1
+    cur_header_hl_group = header_hl_groups[cur_group]
+    hi_range(i, 0, l:len(), cur_header_hl_group)
+  elseif l:find('^## (.+)$') ~= nil then
+    -- Header 2
+    hi_range(i, 0, l:len(), cur_header_hl_group)
+  elseif cur_info ~= nil then
+    -- Plugin info
+    local end_col = l:match('(). +%b()$') or l:len()
+    hi_range(i, cur_info:len(), end_col, 'DiagnosticInfo')
+
+    -- Plugin state after update
+    local col = l:match('() %b()$') or l:len()
+    hi_range(i, col, l:len(), 'DiagnosticHint')
+  elseif l:match('^> ') then
+    -- Added change with possibly "breaking message"
+    hi_range(i, 0, l:len(), 'Added')
+    local col = l:match('│() %S+!:') or l:match('│() %S+%b()!:') or l:len()
+    hi_range(i, col, l:len(), 'DiagnosticWarn', priority + 1)
+  elseif l:match('^< ') then
+    -- Removed change
+    hi_range(i, 0, l:len(), 'Removed')
+  elseif l:match('^• ') then
+    -- Available newer tags
+    hi_range(i, 4, l:len(), 'DiagnosticHint')
+  end
+end

runtime/lua/vim/_editor.lua

@@ -39,6 +39,7 @@ for k, v in pairs({
   health = true,
   secure = true,
   snippet = true,
+  pack = true,
   _watch = true,
 }) do
   vim._submodules[k] = v

runtime/lua/vim/_meta/options.lua

@@ -2137,6 +2137,8 @@ vim.go.ei = vim.go.eventignore
 --- 	`MenuPopup`,
 --- 	`ModeChanged`,
 --- 	`OptionSet`,
+--- 	`PackChanged`,
+--- 	`PackChangedPre`,
 --- 	`QuickFixCmdPost`,
 --- 	`QuickFixCmdPre`,
 --- 	`QuitPre`,

runtime/lua/vim/health/health.lua

@@ -409,6 +409,25 @@ local function check_external_tools()
   else
     health.warn('ripgrep not available')
   end
+
+  -- `vim.pack` requires `git` executable with version at least 2.36
+  if vim.fn.executable('git') == 1 then
+    local git = vim.fn.exepath('git')
+    local out = vim.system({ 'git', 'version' }, {}):wait().stdout or ''
+    local version = vim.version.parse(out)
+    if version < vim.version.parse('2.36') then
+      local msg = string.format(
+        'git is available (%s), but needs at least version 2.36 (not %s) to work with `vim.pack`',
+        git,
+        tostring(version)
+      )
+      health.warn(msg)
+    else
+      health.ok(('%s (%s)'):format(vim.trim(out), git))
+    end
+  else
+    health.warn('git not available (required by `vim.pack`)')
+  end
 end
 
 function M.check()

runtime/lua/vim/pack/_lsp.lua

@@ -0,0 +1,168 @@
+local M = {}
+
+local capabilities = {
+  codeActionProvider = true,
+  documentSymbolProvider = true,
+  hoverProvider = true,
+}
+--- @type table<string,function>
+local methods = {}
+
+--- @param callback function
+function methods.initialize(_, callback)
+  return callback(nil, { capabilities = capabilities })
+end
+
+--- @param callback function
+function methods.shutdown(_, callback)
+  return callback(nil, nil)
+end
+
+local get_confirm_bufnr = function(uri)
+  return tonumber(uri:match('^nvim%-pack://(%d+)/confirm%-update$'))
+end
+
+--- @param params { textDocument: { uri: string } }
+--- @param callback function
+methods['textDocument/documentSymbol'] = function(params, callback)
+  local bufnr = get_confirm_bufnr(params.textDocument.uri)
+  if bufnr == nil then
+    return callback(nil, {})
+  end
+
+  --- @alias vim.pack.lsp.Position { line: integer, character: integer }
+  --- @alias vim.pack.lsp.Range { start: vim.pack.lsp.Position, end: vim.pack.lsp.Position }
+  --- @alias vim.pack.lsp.Symbol {
+  ---   name: string,
+  ---   kind: number,
+  ---   range: vim.pack.lsp.Range,
+  ---   selectionRange: vim.pack.lsp.Range,
+  ---   children: vim.pack.lsp.Symbol[]?,
+  --- }
+
+  --- @return vim.pack.lsp.Symbol?
+  local new_symbol = function(name, start_line, end_line, kind)
+    if name == nil then
+      return nil
+    end
+    local range = {
+      start = { line = start_line, character = 0 },
+      ['end'] = { line = end_line, character = 0 },
+    }
+    return { name = name, kind = kind, range = range, selectionRange = range }
+  end
+
+  local lines = vim.api.nvim_buf_get_lines(bufnr, 0, -1, false)
+
+  --- @return vim.pack.lsp.Symbol[]
+  local parse_headers = function(pattern, start_line, end_line, kind)
+    local res, cur_match, cur_start = {}, nil, nil
+    for i = start_line, end_line do
+      local m = lines[i + 1]:match(pattern)
+      if m ~= nil and m ~= cur_match then
+        table.insert(res, new_symbol(cur_match, cur_start, i, kind))
+        cur_match, cur_start = m, i
+      end
+    end
+    table.insert(res, new_symbol(cur_match, cur_start, end_line, kind))
+    return res
+  end
+
+  local group_kind = vim.lsp.protocol.SymbolKind.Namespace
+  local symbols = parse_headers('^# (%S+)', 0, #lines - 1, group_kind)
+
+  local plug_kind = vim.lsp.protocol.SymbolKind.Module
+  for _, group in ipairs(symbols) do
+    local start_line, end_line = group.range.start.line, group.range['end'].line
+    group.children = parse_headers('^## (.+)$', start_line, end_line, plug_kind)
+  end
+
+  return callback(nil, symbols)
+end
+
+--- @param callback function
+methods['textDocument/codeAction'] = function(_, callback)
+  -- TODO(echasnovski)
+  -- Suggested actions for "plugin under cursor":
+  -- - Delete plugin from disk.
+  -- - Update only this plugin.
+  -- - Exclude this plugin from update.
+  return callback(_, {})
+end
+
+--- @param params { textDocument: { uri: string }, position: { line: integer, character: integer } }
+--- @param callback function
+methods['textDocument/hover'] = function(params, callback)
+  local bufnr = get_confirm_bufnr(params.textDocument.uri)
+  if bufnr == nil then
+    return
+  end
+
+  local lines = vim.api.nvim_buf_get_lines(bufnr, 0, -1, false)
+  local lnum = params.position.line + 1
+  local commit = lines[lnum]:match('^[<>] (%x+) │') or lines[lnum]:match('^State.*:%s+(%x+)')
+  local tag = lines[lnum]:match('^• (.+)$')
+  if commit == nil and tag == nil then
+    return
+  end
+
+  local path, path_lnum = nil, lnum - 1
+  while path == nil and path_lnum >= 1 do
+    path = lines[path_lnum]:match('^Path:%s+(.+)$')
+    path_lnum = path_lnum - 1
+  end
+  if path == nil then
+    return
+  end
+
+  local cmd = { 'git', 'show', '--no-color', commit or tag }
+  --- @param sys_out vim.SystemCompleted
+  local on_exit = function(sys_out)
+    local markdown = '```diff\n' .. sys_out.stdout .. '\n```'
+    local res = { contents = { kind = vim.lsp.protocol.MarkupKind.Markdown, value = markdown } }
+    callback(nil, res)
+  end
+  vim.system(cmd, { cwd = path }, vim.schedule_wrap(on_exit))
+end
+
+local dispatchers = {}
+
+-- TODO: Simplify after `vim.lsp.server` is a thing
+-- https://github.com/neovim/neovim/pull/24338
+local cmd = function(disp)
+  -- Store dispatchers to use for showing progress notifications
+  dispatchers = disp
+  local res, closing, request_id = {}, false, 0
+
+  function res.request(method, params, callback)
+    local method_impl = methods[method]
+    if method_impl ~= nil then
+      method_impl(params, callback)
+    end
+    request_id = request_id + 1
+    return true, request_id
+  end
+
+  function res.notify(method, _)
+    if method == 'exit' then
+      dispatchers.on_exit(0, 15)
+    end
+    return false
+  end
+
+  function res.is_closing()
+    return closing
+  end
+
+  function res.terminate()
+    closing = true
+  end
+
+  return res
+end
+
+M.client_id = assert(
+  vim.lsp.start({ cmd = cmd, name = 'vim.pack', root_dir = vim.uv.cwd() }, { attach = false })
+)
+
+return M

src/gen/gen_vimdoc.lua

@@ -144,6 +144,7 @@ local config = {
       '_inspector.lua',
       'shared.lua',
       'loader.lua',
+      'pack.lua',
       'uri.lua',
       'ui.lua',
       '_extui.lua',
@@ -167,6 +168,7 @@ local config = {
       'runtime/lua/vim/_options.lua',
       'runtime/lua/vim/shared.lua',
       'runtime/lua/vim/loader.lua',
+      'runtime/lua/vim/pack.lua',
       'runtime/lua/vim/uri.lua',
       'runtime/lua/vim/ui.lua',
       'runtime/lua/vim/_extui.lua',

src/nvim/auevents.lua

@@ -87,6 +87,8 @@ return {
     QuickFixCmdPost = false, -- after :make, :grep etc.
     QuickFixCmdPre = false, -- before :make, :grep etc.
     QuitPre = false, -- before :quit
+    PackChangedPre = false, -- before trying to change state of `vim.pack` plugin
+    PackChanged = false, -- after changing state of `vim.pack` plugin
     RecordingEnter = true, -- when starting to record a macro
     RecordingLeave = true, -- just before a macro stops recording
     RemoteReply = false, -- upon string reception from a remote vim
@@ -158,6 +160,8 @@ return {
     LspProgress = true,
     LspRequest = true,
     LspTokenUpdate = true,
+    PackChangedPre = true,
+    PackChanged = true,
     RecordingEnter = true,
     RecordingLeave = true,
     Signal = true,

test/functional/plugin/pack_spec.lua

@@ -0,0 +1,68 @@
+describe('vim.pack', function()
+  describe('add()', function()
+    pending('works', function()
+      -- TODO
+    end)
+
+    pending('respects after/', function()
+      -- TODO
+      -- Should source 'after/plugin/' directory (even nested files) after
+      -- all 'plugin/' files are sourced in all plugins from input.
+      --
+      -- Should add 'after/' directory (if present) to 'runtimepath'
+    end)
+
+    pending('normalizes each spec', function()
+      -- TODO
+
+      -- TODO: Should properly infer `name` from `src` (as its basename
+      -- minus '.git' suffix) but allow '.git' suffix in explicit `name`
+    end)
+
+    pending('normalizes spec array', function()
+      -- TODO
+      -- Should silently ignore full duplicates (same `src`+`version`)
+      -- and error on conflicts.
+    end)
+
+    pending('installs', function()
+      -- TODO
+
+      -- TODO: Should block code flow until all plugins are available on disk
+      -- and `:packadd` all of them (even just now installed) as a result.
+    end)
+  end)
+
+  describe('update()', function()
+    pending('works', function()
+      -- TODO
+
+      -- TODO: Should work with both added and not added plugins
+    end)
+
+    pending('suggests newer tags if there are no updates', function()
+      -- TODO
+
+      -- TODO: Should not suggest tags that point to the current state.
+      -- Even if there is one/several and located at start/middle/end.
+    end)
+  end)
+
+  describe('get()', function()
+    pending('works', function()
+      -- TODO
+    end)
+
+    pending('works after `del()`', function()
+      -- TODO: Should not include removed plugins and still return list
+
+      -- TODO: Should return corrent list inside `PackChanged` "delete" event
+    end)
+  end)
+
+  describe('del()', function()
+    pending('works', function()
+      -- TODO
+    end)
+  end)
+end)